You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Johnny X <jo...@gmail.com> on 2009/04/11 02:36:53 UTC
PHP Remove From Index/Search By Fields
Hey,
How could I write some code in PHP to place in a button to remove a returned
item from the index?
In turn, is it possible to copy all of the XML elements from said item and
place them in a document somewhere locally once it's been removed?
Finally, there is one default search field. How do you search on multiple
different fields in PHP?
If I wanted to search by all of the fields indexed, is that easy to code?
What changes do I need to make in the XML schema?
Thanks for much for any help!
--
View this message in context: http://www.nabble.com/PHP-Remove-From-Index-Search-By-Fields-tp22996701p22996701.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PHP Remove From Index/Search By Fields
Posted by Johnny X <jo...@gmail.com>.
A further update on this is that (when 'Date' is searched using the same URL
as posted in the previous message), whether Date is of type string or text,
the full (exact) content of a field has to be searched to return a result.
Why is this not the case with Content? I tried changing the default search
field to 'Date' to see if that made a difference and nothing changed.
Johnny X wrote:
>
> Do you know the specific syntax when querying different fields?
>
> http://localhost:8080/solr/select/?q=Date:%222000%22&version=2.2&start=0&rows=10&indent=on
>
> doesn't appear to return anything when I post it in my browser, when it
> should, but (as before) if you change 'Date' to 'Content' it works!
> (presumably because content is the default field). Is there nothing else I
> have to change to make sure they're returned? All fields are indexed and
> stored, but 'Content' is the only 'text' field, the others are 'string'.
>
> Going back to dismax, it looks like that's more useful for boosting than
> specifying multiple fields because it works a lot like copyfields (in that
> it compounds all of the fields together in one big search). If I were to
> do that, there'd be no need to have anything more than one user input box
> because it won't be separated by field anyway.
>
>
>
> Erik Hatcher wrote:
>>
>>
>> On Apr 13, 2009, at 11:20 AM, Johnny X wrote:
>>
>>>
>>> Also, in reference to the other question, I'm currently trying to
>>> edit the
>>> main search page to search multiple fields.
>>>
>>> Essentially, I detect if each field has been posted or not using:
>>>
>>> if ($_POST['FIELD'] != '') {
>>> $query = $query . '+FIELDNAME:' . $_POST['FIELD'];
>>> }
>>>
>>> Once it's processed all the fields, its then sent to query solr, but
>>> I'm not
>>> sure if I'm getting the syntax right or if there's anything in the
>>> Solr
>>> config file I need to modify (dismax?) because it still only returns
>>> results
>>> when I enter a search in the 'content' field (also the default Solr
>>> field).
>>>
>>> My Solr query looks like:
>>>
>>> $query = "?q=".trim(urlencode($query)).
>>> '&version=2.2&start=0&rows=999999&indent=on';
>>>
>>> where $query will look something like "Content: 35 million+Date:
>>> 16th Oct"
>>> etc, until it has been urlencoded/trimmed.
>>>
>>> Will it still only return results on 'content' searches because
>>> that's the
>>> only default field?
>>
>> You'll need to read up on Lucene/Solr query parser syntax to be able
>> to build useful queries programatically like that:
>> <http://wiki.apache.org/solr/SolrQuerySyntax
>> > Your syntax above is not doing what you might think... you'll want
>> to surround expressions with quotes or in parens for a single field.
>> Content:(35 million) for example.
>>
>> It'll be best if you decouple your questions about query parsing from
>> PHP code though. And don't forget that &debugQuery=true is your
>> friend, so you can see how queries are being parsed. Providing that
>> output would be helpful to see what is actually happening with what
>> you're sending.
>>
>> Erik
>>
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PHP-Remove-From-Index-Search-By-Fields-tp22996701p23025514.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PHP Remove From Index/Search By Fields
Posted by Johnny X <jo...@gmail.com>.
Do you know the specific syntax when querying different fields?
http://localhost:8080/solr/select/?q=Date:%222000%22&version=2.2&start=0&rows=10&indent=on
doesn't appear to return anything when I post it in my browser, when it
should, but (as before) if you change 'Date' to 'Content' it works!
(presumably because content is the default field). Is there nothing else I
have to change to make sure they're returned? All fields are indexed and
stored, but 'Content' is the only 'text' field, the others are 'string'.
Going back to dismax, it looks like that's more useful for boosting than
specifying multiple fields because it works a lot like copyfields (in that
it compounds all of the fields together in one big search). If I were to do
that, there'd be no need to have anything more than one user input box
because it won't be separated by field anyway.
Erik Hatcher wrote:
>
>
> On Apr 13, 2009, at 11:20 AM, Johnny X wrote:
>
>>
>> Also, in reference to the other question, I'm currently trying to
>> edit the
>> main search page to search multiple fields.
>>
>> Essentially, I detect if each field has been posted or not using:
>>
>> if ($_POST['FIELD'] != '') {
>> $query = $query . '+FIELDNAME:' . $_POST['FIELD'];
>> }
>>
>> Once it's processed all the fields, its then sent to query solr, but
>> I'm not
>> sure if I'm getting the syntax right or if there's anything in the
>> Solr
>> config file I need to modify (dismax?) because it still only returns
>> results
>> when I enter a search in the 'content' field (also the default Solr
>> field).
>>
>> My Solr query looks like:
>>
>> $query = "?q=".trim(urlencode($query)).
>> '&version=2.2&start=0&rows=999999&indent=on';
>>
>> where $query will look something like "Content: 35 million+Date:
>> 16th Oct"
>> etc, until it has been urlencoded/trimmed.
>>
>> Will it still only return results on 'content' searches because
>> that's the
>> only default field?
>
> You'll need to read up on Lucene/Solr query parser syntax to be able
> to build useful queries programatically like that:
> <http://wiki.apache.org/solr/SolrQuerySyntax
> > Your syntax above is not doing what you might think... you'll want
> to surround expressions with quotes or in parens for a single field.
> Content:(35 million) for example.
>
> It'll be best if you decouple your questions about query parsing from
> PHP code though. And don't forget that &debugQuery=true is your
> friend, so you can see how queries are being parsed. Providing that
> output would be helpful to see what is actually happening with what
> you're sending.
>
> Erik
>
>
>
>
--
View this message in context: http://www.nabble.com/PHP-Remove-From-Index-Search-By-Fields-tp22996701p23024816.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PHP Remove From Index/Search By Fields
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Apr 13, 2009, at 11:20 AM, Johnny X wrote:
>
> Also, in reference to the other question, I'm currently trying to
> edit the
> main search page to search multiple fields.
>
> Essentially, I detect if each field has been posted or not using:
>
> if ($_POST['FIELD'] != '') {
> $query = $query . '+FIELDNAME:' . $_POST['FIELD'];
> }
>
> Once it's processed all the fields, its then sent to query solr, but
> I'm not
> sure if I'm getting the syntax right or if there's anything in the
> Solr
> config file I need to modify (dismax?) because it still only returns
> results
> when I enter a search in the 'content' field (also the default Solr
> field).
>
> My Solr query looks like:
>
> $query = "?q=".trim(urlencode($query)).
> '&version=2.2&start=0&rows=999999&indent=on';
>
> where $query will look something like "Content: 35 million+Date:
> 16th Oct"
> etc, until it has been urlencoded/trimmed.
>
> Will it still only return results on 'content' searches because
> that's the
> only default field?
You'll need to read up on Lucene/Solr query parser syntax to be able
to build useful queries programatically like that: <http://wiki.apache.org/solr/SolrQuerySyntax
> Your syntax above is not doing what you might think... you'll want
to surround expressions with quotes or in parens for a single field.
Content:(35 million) for example.
It'll be best if you decouple your questions about query parsing from
PHP code though. And don't forget that &debugQuery=true is your
friend, so you can see how queries are being parsed. Providing that
output would be helpful to see what is actually happening with what
you're sending.
Erik
Re: PHP Remove From Index/Search By Fields
Posted by Johnny X <jo...@gmail.com>.
Also, in reference to the other question, I'm currently trying to edit the
main search page to search multiple fields.
Essentially, I detect if each field has been posted or not using:
if ($_POST['FIELD'] != '') {
$query = $query . '+FIELDNAME:' . $_POST['FIELD'];
}
Once it's processed all the fields, its then sent to query solr, but I'm not
sure if I'm getting the syntax right or if there's anything in the Solr
config file I need to modify (dismax?) because it still only returns results
when I enter a search in the 'content' field (also the default Solr field).
My Solr query looks like:
$query = "?q=".trim(urlencode($query)).
'&version=2.2&start=0&rows=999999&indent=on';
where $query will look something like "Content: 35 million+Date: 16th Oct"
etc, until it has been urlencoded/trimmed.
Will it still only return results on 'content' searches because that's the
only default field?
Johnny X wrote:
>
> Thanks for the reply Erik!
>
> Based on a previous page I used to return queries I've developed this code
> below for the page I need to do all of the above.
>
> CODE
>
> <?php
>
>
> $id = $_GET['id'];
>
> $connection = mysqli_connect("localhost", "root", "onion", "collection")
> or die ("Couldn't connect to MySQL");
>
> define('SOLR_URL', 'http://localhost:8080/solr/');
>
> function request($reqData, $type){
>
> $header[] = "Content-type: text/xml; charset=UTF-8";
>
> $session = curl_init();
> curl_setopt($session, CURLOPT_HEADER, true);
> curl_setopt($session, CURLOPT_HTTPHEADER, $header);
> curl_setopt($session, CURLOPT_URL, SOLR_URL.$type);
> curl_setopt($session, CURLOPT_POSTFIELDS, $reqData);
> curl_setopt($session, CURLOPT_RETURNTRANSFER, 1);
> curl_setopt($session, CURLOPT_POST, 1);
>
> $response = curl_exec($session);
> curl_close($session);
>
> return $response;
> }
>
> function solrQuery($q){
> $query =
> "?q=".trim(urlencode($q))."qf=Message-ID&version=2.2&start=0&rows=999999&indent=on";
> return $results = request("", "select".$query);
>
> }
>
> echo "<html><head><title>IP E-mail</title>";
> echo '<link rel="stylesheet" type="text/css" href="stylesheet.css" />';
>
> echo '<script type="text/javascript">
> <!--
> function confirmation() {
> var answer = confirm("Remove spam?")
> if (answer){
> alert("Spam removed!")
> $results = solrQuery('.$id.');
> }
>
> }
> //-->
> </script>';
>
>
> echo "</head><body>";
>
>
> echo '<form method="post">';
> echo '<table width="100%">';
> echo "<tr>";
> echo '<td><h1>Trace/Mark IP E-mail</h1><td>';
> echo '<td><p align="right">Powered by</p></td>';
> echo '<td width="283px"> mysql_logo.jpg </td>';
> echo "</tr>";
> echo "</table>";
> echo "</form>";
>
> /* Send a query to the server */
> if ($location = mysqli_query($connection, "SELECT location FROM hashes
> WHERE message_id = '$id'")) {
>
> echo '<br/>';
> echo '<p>Mark as: <input type="button" onclick="confirmation()"
> value="Spam"> <input type="button" value="Non-Business"> <input
> type="button" value="Non-Confidential"></p>';
>
> print("<h3>Message Location:\n</h3>");
>
> /* Fetch the results of the query */
> while( $row = mysqli_fetch_assoc($location) ){
> printf("<p>%s\n</p>", $row['location']);
> }
>
> /* Destroy the result set and free the memory used for it */
> mysqli_free_result($location);
> }
>
>
> /* Send a query to the server */
> if ($duplicates = mysqli_query($connection, "SELECT location FROM hashes
> WHERE (md5 = (SELECT md5 FROM hashes WHERE message_id = '$id') AND
> message_id <> '$id')")) {
>
> print("<h3>Duplicate Locations:\n</h3>");
>
> /* Fetch the results of the query */
> while( $row = mysqli_fetch_assoc($duplicates) ){
> printf("<p>%s\n</p>", $row['location']);
> }
>
> /* Destroy the result set and free the memory used for it */
> mysqli_free_result($duplicates);
> }
>
>
>
>
> /* Close the connection */
> mysqli_close($connection);
>
>
>
>
>
>
>
> $results =
> explode('<?xml version="1.0" encoding="UTF-8"?>', $results);
> $results = $results[1];
>
> $dom = new DomDocument;
> $dom->loadXML($results);
> $docs = $dom->getElementsByTagName('doc');
>
> foreach ($docs as $doc) {
> $strings = $doc->getElementsByTagName('arr');
> foreach($strings as $str){
> $attr = $str->getAttribute('name');
> $data = $str->textContent;
> switch($attr){
> case 'Bcc':
> $Bcc = $data;
> break;
> case 'Cc':
> $Cc = $data;
> break;
> case 'Content':
> $Content = $data;
> break;
> case 'Content-Transfer-Encoding':
> $ContentTransferEncoding = $data;
> break;
> case 'Content-Type':
> $ContentType = $data;
> break;
> case 'Date':
> $Date = $data;
> break;
> case 'From':
> $From = $data;
> break;
> case 'Message-ID':
> $MessageID = $data;
> break;
> case 'Mime-Version':
> $MimeVersion = $data;
> break;
> case 'Subject':
> $Subject = $data;
> break;
> case 'To':
> $To = $data;
> break;
> case 'X-FileName':
> $XFileName = $data;
> break;
> case 'X-Folder':
> $XFolder = $data;
> break;
> case 'X-From':
> $XFrom = $data;
> break;
> case 'X-Origin':
> $XOrigin = $data;
> break;
> case 'X-To':
> $XTo = $data;
> break;
> case 'X-bcc':
> $Xbcc = $data;
> break;
> case 'X-cc':
> $Xcc = $data;
> break;
> }
> }
>
> }
>
>
> echo "</body></html>";
>
>
>
>
> ?>
> /CODE
>
>
> The issue I'm having is working out how to use this to submit a POST
> delete Solr query on a button click, or how to use the document (once
> returned, like the query page) and save it in text form to disk
> (essentially, excluding all XML mark-up)...is this done using Curl
> (supported)...or AJAX?
>
>
> Any ideas?
>
>
> Cheers.
>
>
>
>
>
> Erik Hatcher wrote:
>>
>>
>> On Apr 10, 2009, at 8:36 PM, Johnny X wrote:
>>> How could I write some code in PHP to place in a button to remove a
>>> returned
>>> item from the index?
>>
>> You can issue a delete command to Solr by simply doing a GET (or POST)
>> to
>> http://localhost:8983/solr/update?stream.body=%3Cdelete%3E%3Cid%3ESOMEID%3C/id%3E%3C/delete%3Eete%3E%3Cquery%3Esome:query%3C/query%3E%3C/delete%3E&commit=true
>>
>> (replace SOMEID with whatever unique id you want, or switch to use the
>> delete-by-query)
>>
>>> In turn, is it possible to copy all of the XML elements from said
>>> item and
>>> place them in a document somewhere locally once it's been removed?
>>
>> You could do a /select?q=id:SOMEID&fl=* and get all stored field
>> values (note you'll lose any unstored values) and stash those results
>> off however you like.
>>
>>> Finally, there is one default search field. How do you search on
>>> multiple
>>> different fields in PHP?
>>
>> Use the dismax parser. /select?
>> q=whatever&defType=dismax&qf=field1+field2
>>
>> See the wiki docs on dismax for more details and list of other
>> parameters.
>>
>>> If I wanted to search by all of the fields indexed, is that easy to
>>> code?
>>
>> You could list all fields in the qf field of dismax.
>>
>> Another option is to copyField all fields you want searchable into a
>> single field and search that single field.
>>
>>> What changes do I need to make in the XML schema?
>>
>> To use dismax, no changes to schema are needed (necessarily).
>>
>> But you may want to add copyField's to schema.xml to collect all field
>> text into a single field for default field searchability.
>>
>> Erik
>>
>>
>
>
--
View this message in context: http://www.nabble.com/PHP-Remove-From-Index-Search-By-Fields-tp22996701p23023747.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PHP Remove From Index/Search By Fields
Posted by Johnny X <jo...@gmail.com>.
Thanks for the reply Erik!
Based on a previous page I used to return queries I've developed this code
below for the page I need to do all of the above.
CODE
<?php
$id = $_GET['id'];
$connection = mysqli_connect("localhost", "root", "onion", "collection") or
die ("Couldn't connect to MySQL");
define('SOLR_URL', 'http://localhost:8080/solr/');
function request($reqData, $type){
$header[] = "Content-type: text/xml; charset=UTF-8";
$session = curl_init();
curl_setopt($session, CURLOPT_HEADER, true);
curl_setopt($session, CURLOPT_HTTPHEADER, $header);
curl_setopt($session, CURLOPT_URL, SOLR_URL.$type);
curl_setopt($session, CURLOPT_POSTFIELDS, $reqData);
curl_setopt($session, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($session, CURLOPT_POST, 1);
$response = curl_exec($session);
curl_close($session);
return $response;
}
function solrQuery($q){
$query =
"?q=".trim(urlencode($q))."qf=Message-ID&version=2.2&start=0&rows=999999&indent=on";
return $results = request("", "select".$query);
}
echo "<html><head><title>IP E-mail</title>";
echo '<link rel="stylesheet" type="text/css" href="stylesheet.css" />';
echo '<script type="text/javascript">
<!--
function confirmation() {
var answer = confirm("Remove spam?")
if (answer){
alert("Spam removed!")
$results = solrQuery('.$id.');
}
}
//-->
</script>';
echo "</head><body>";
echo '<form method="post">';
echo '<table width="100%">';
echo "<tr>";
echo '<td><h1>Trace/Mark IP E-mail</h1><td>';
echo '<td><p align="right">Powered by</p></td>';
echo '<td width="283px"> mysql_logo.jpg </td>';
echo "</tr>";
echo "</table>";
echo "</form>";
/* Send a query to the server */
if ($location = mysqli_query($connection, "SELECT location FROM hashes WHERE
message_id = '$id'")) {
echo '<br/>';
echo '<p>Mark as: <input type="button" onclick="confirmation()"
value="Spam"> <input type="button" value="Non-Business"> <input
type="button" value="Non-Confidential"></p>';
print("<h3>Message Location:\n</h3>");
/* Fetch the results of the query */
while( $row = mysqli_fetch_assoc($location) ){
printf("<p>%s\n</p>", $row['location']);
}
/* Destroy the result set and free the memory used for it */
mysqli_free_result($location);
}
/* Send a query to the server */
if ($duplicates = mysqli_query($connection, "SELECT location FROM hashes
WHERE (md5 = (SELECT md5 FROM hashes WHERE message_id = '$id') AND
message_id <> '$id')")) {
print("<h3>Duplicate Locations:\n</h3>");
/* Fetch the results of the query */
while( $row = mysqli_fetch_assoc($duplicates) ){
printf("<p>%s\n</p>", $row['location']);
}
/* Destroy the result set and free the memory used for it */
mysqli_free_result($duplicates);
}
/* Close the connection */
mysqli_close($connection);
$results =
explode('<?xml version="1.0" encoding="UTF-8"?>', $results);
$results = $results[1];
$dom = new DomDocument;
$dom->loadXML($results);
$docs = $dom->getElementsByTagName('doc');
foreach ($docs as $doc) {
$strings = $doc->getElementsByTagName('arr');
foreach($strings as $str){
$attr = $str->getAttribute('name');
$data = $str->textContent;
switch($attr){
case 'Bcc':
$Bcc = $data;
break;
case 'Cc':
$Cc = $data;
break;
case 'Content':
$Content = $data;
break;
case 'Content-Transfer-Encoding':
$ContentTransferEncoding = $data;
break;
case 'Content-Type':
$ContentType = $data;
break;
case 'Date':
$Date = $data;
break;
case 'From':
$From = $data;
break;
case 'Message-ID':
$MessageID = $data;
break;
case 'Mime-Version':
$MimeVersion = $data;
break;
case 'Subject':
$Subject = $data;
break;
case 'To':
$To = $data;
break;
case 'X-FileName':
$XFileName = $data;
break;
case 'X-Folder':
$XFolder = $data;
break;
case 'X-From':
$XFrom = $data;
break;
case 'X-Origin':
$XOrigin = $data;
break;
case 'X-To':
$XTo = $data;
break;
case 'X-bcc':
$Xbcc = $data;
break;
case 'X-cc':
$Xcc = $data;
break;
}
}
}
echo "</body></html>";
?>
/CODE
The issue I'm having is working out how to use this to submit a POST delete
Solr query on a button click, or how to use the document (once returned,
like the query page) and save it in text form to disk (essentially,
excluding all XML mark-up)...is this done using Curl (supported)...or AJAX?
Any ideas?
Cheers.
Erik Hatcher wrote:
>
>
> On Apr 10, 2009, at 8:36 PM, Johnny X wrote:
>> How could I write some code in PHP to place in a button to remove a
>> returned
>> item from the index?
>
> You can issue a delete command to Solr by simply doing a GET (or POST)
> to
> http://localhost:8983/solr/update?stream.body=%3Cdelete%3E%3Cid%3ESOMEID%3C/id%3E%3C/delete%3Eete%3E%3Cquery%3Esome:query%3C/query%3E%3C/delete%3E&commit=true
>
> (replace SOMEID with whatever unique id you want, or switch to use the
> delete-by-query)
>
>> In turn, is it possible to copy all of the XML elements from said
>> item and
>> place them in a document somewhere locally once it's been removed?
>
> You could do a /select?q=id:SOMEID&fl=* and get all stored field
> values (note you'll lose any unstored values) and stash those results
> off however you like.
>
>> Finally, there is one default search field. How do you search on
>> multiple
>> different fields in PHP?
>
> Use the dismax parser. /select?
> q=whatever&defType=dismax&qf=field1+field2
>
> See the wiki docs on dismax for more details and list of other
> parameters.
>
>> If I wanted to search by all of the fields indexed, is that easy to
>> code?
>
> You could list all fields in the qf field of dismax.
>
> Another option is to copyField all fields you want searchable into a
> single field and search that single field.
>
>> What changes do I need to make in the XML schema?
>
> To use dismax, no changes to schema are needed (necessarily).
>
> But you may want to add copyField's to schema.xml to collect all field
> text into a single field for default field searchability.
>
> Erik
>
>
--
View this message in context: http://www.nabble.com/PHP-Remove-From-Index-Search-By-Fields-tp22996701p23016677.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: PHP Remove From Index/Search By Fields
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Apr 10, 2009, at 8:36 PM, Johnny X wrote:
> How could I write some code in PHP to place in a button to remove a
> returned
> item from the index?
You can issue a delete command to Solr by simply doing a GET (or POST)
to http://localhost:8983/solr/update?stream.body=%3Cdelete%3E%3Cid%3ESOMEID%3C/id%3E%3C/delete%3Eete%3E%3Cquery%3Esome:query%3C/query%3E%3C/delete%3E&commit=true
(replace SOMEID with whatever unique id you want, or switch to use the
delete-by-query)
> In turn, is it possible to copy all of the XML elements from said
> item and
> place them in a document somewhere locally once it's been removed?
You could do a /select?q=id:SOMEID&fl=* and get all stored field
values (note you'll lose any unstored values) and stash those results
off however you like.
> Finally, there is one default search field. How do you search on
> multiple
> different fields in PHP?
Use the dismax parser. /select?
q=whatever&defType=dismax&qf=field1+field2
See the wiki docs on dismax for more details and list of other
parameters.
> If I wanted to search by all of the fields indexed, is that easy to
> code?
You could list all fields in the qf field of dismax.
Another option is to copyField all fields you want searchable into a
single field and search that single field.
> What changes do I need to make in the XML schema?
To use dismax, no changes to schema are needed (necessarily).
But you may want to add copyField's to schema.xml to collect all field
text into a single field for default field searchability.
Erik