You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Paul Sundling (JIRA)" <ji...@apache.org> on 2007/08/04 02:49:52 UTC

[jira] Created: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
-----------------------------------------------------------------------------

                 Key: SOLR-327
                 URL: https://issues.apache.org/jira/browse/SOLR-327
             Project: Solr
          Issue Type: Bug
            Reporter: Paul Sundling
            Priority: Minor


There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.

The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)

There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Re: [jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by Bill Au <bi...@gmail.com>.
The "date -d" problem on Solaris has also been fixed.  See:

http://issues.apache.org/jira/browse/SOLR-294

Bill

On 8/8/07, Ken Prows (JIRA) <ji...@apache.org> wrote:
>
>     [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518544 ]
>
> Ken Prows commented on SOLR-327:
> --------------------------------
>
> On Solaris 10 I also ran into a problem with the  date command. The -d option does not exist for the Solaris version of date. This causes the snappuller script to fail. In order to fix this, I made the below changes. Note that I made these changes off of the latest version from trunk (revision 556910) since there were already some Solaris fixes there.
>
> CHANGED:
>
> rsyncStart=`date +'%Y-%m-%d %H:%M:%S'`
> startTimestamp=`date -d "$rsyncStart" +'%Y%m%d-%H%M%S'`
> rsyncStartSec=`date -d "$rsyncStart" +'%s'`
>
> TO:
>
> rsyncStartSec=`perl -e 'print time'`
> startTimestamp=`perl -MPOSIX -e 'print POSIX::strftime("%Y%m%d%H%M%S", localtime(time))'`
>
>
> CHANGED:
>
> rsyncEnd=`date +'%Y-%m-%d %H:%M:%S'`
> endTimestamp=`date -d "$rsyncEnd" +'%Y%m%d-%H%M%S'`
> rsyncEndSec=`date -d "$rsyncEnd" +'%s'`
>
> TO:
>
> rsyncEndSec=`perl -e 'print time'`
> endTimestamp=`perl -MPOSIX -e 'print POSIX::strftime("%Y%m%d%H%M%S", localtime(time))'`
>
>
> ----
>
> As a side note, I'd like to see the  scripts converted from bash to perl for better portability. If there is any interest in that, I'd like to help take that on.
>
>
> > shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> > -----------------------------------------------------------------------------
> >
> >                 Key: SOLR-327
> >                 URL: https://issues.apache.org/jira/browse/SOLR-327
> >             Project: Solr
> >          Issue Type: Bug
> >            Reporter: Paul Sundling
> >            Priority: Minor
> >
> > There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> > The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> > There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Ken Prows (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518815 ] 

Ken Prows commented on SOLR-327:
--------------------------------

On second thought, cygwin may be required to get the scripts to work in windows because of the ssh/rsync requirement. 

I've read that you can already get the current bash scripts to run under cygwin, so not much would change on the windows front.

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518822 ] 

Yonik Seeley commented on SOLR-327:
-----------------------------------

Hard links work on NTFS, but the differing semantics still mess things up.
For example, you can't delete *any* of the links to an open file.

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519189 ] 

Hoss Man commented on SOLR-327:
-------------------------------

the replication scripts are not 100% portable on every OS out there -- but i don't know that they ever can be -- regardless of what language they are written in, or what filesystems are used.

the important thing is that they are abstracted from the Solr java code in a way that makes it possible for people with funky OSes/filesystems to implement any script/application they want and register it as a postCommit or postOptimize hook on a master box (and send a commit on slave machines when it's time to open a freshly synced index)

if the bash scripts work for a lot of people -- GREAT! ... if people want to provide perl scripts that do similar things, also great; if someone wants to submit C code for compiling native binary applications that take care of replication, that's great too.

the scripts we have now exist because they worked on the platform the original Solr developers used, that doesn't mean there can't be other completley different replication scripts/apps that work on other platforms (or work in completely different ways)  

I personally am more concerned with making sure we don't break the scripts on any platforms they currently do work on then i am about making them work on new platforms.

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Paul Sundling (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518574 ] 

Paul Sundling commented on SOLR-327:
------------------------------------

I agree with Ken.  The true issue I was pointing out was the issue of portability, with Solaris 8 being just a single use case.  So a particular command may be an issue on Solaris 8 or Solaris 10 or AIX or whatever.  

The CENTRAL question is do we provide the greatest portability, by a lowest common denominator bash script or use Perl script?  

When I've built Perl in the past, it goes through an extensive check to see the capabilities of the underlying system.  Perl is probably the best bet for portability.  Perl is even available on Windows.

One thing is that with Perl scripts, users on Windows would expect the scripts to work there too.  Would the scripts all work on windows, including hard links?

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated SOLR-327:
--------------------------

    Component/s: replication

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>          Components: replication
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Ken Prows (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518544 ] 

Ken Prows commented on SOLR-327:
--------------------------------

On Solaris 10 I also ran into a problem with the  date command. The -d option does not exist for the Solaris version of date. This causes the snappuller script to fail. In order to fix this, I made the below changes. Note that I made these changes off of the latest version from trunk (revision 556910) since there were already some Solaris fixes there.

CHANGED:

rsyncStart=`date +'%Y-%m-%d %H:%M:%S'`
startTimestamp=`date -d "$rsyncStart" +'%Y%m%d-%H%M%S'`
rsyncStartSec=`date -d "$rsyncStart" +'%s'`

TO:

rsyncStartSec=`perl -e 'print time'`
startTimestamp=`perl -MPOSIX -e 'print POSIX::strftime("%Y%m%d%H%M%S", localtime(time))'`


CHANGED:

rsyncEnd=`date +'%Y-%m-%d %H:%M:%S'`
endTimestamp=`date -d "$rsyncEnd" +'%Y%m%d-%H%M%S'`
rsyncEndSec=`date -d "$rsyncEnd" +'%s'`

TO:

rsyncEndSec=`perl -e 'print time'`
endTimestamp=`perl -MPOSIX -e 'print POSIX::strftime("%Y%m%d%H%M%S", localtime(time))'`


----

As a side note, I'd like to see the  scripts converted from bash to perl for better portability. If there is any interest in that, I'd like to help take that on.


> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Ken Prows (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518810 ] 

Ken Prows commented on SOLR-327:
--------------------------------

I don't think there is anything the scripts do that couldn't be done using the ActivePerl on windows. Hard links work in windows for NTFS volumes.  There are even cpan modules that provide an rsync API.

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-327) shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)

Posted by "Bill Au (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517996 ] 

Bill Au commented on SOLR-327:
------------------------------

The latest version of the scripts no longer use "cp -l" on Solaris:

https://issues.apache.org/jira/browse/SOLR-282

I no longer have access to a Solaris machine so I cannot check myself but according to the Solaris 8 documentation published by SUN pgrep is available on Solaris 8:

http://docs.sun.com/app/docs/doc/806-0624/6j9vek5cu?a=view

What other incompatibilities have you come across?

> shell scripts failed to run on Solaris 8 (and probably other non recent UNIX)
> -----------------------------------------------------------------------------
>
>                 Key: SOLR-327
>                 URL: https://issues.apache.org/jira/browse/SOLR-327
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Paul Sundling
>            Priority: Minor
>
> There are several places where commands used are incompatible with older UNIX versions, even though these capabilities are present.  There are ways to rewrite the shell scripts to be compatible with these older versions.
> The first example is the use of pgrep.  Older machines will have grep, egrep and fgrep, but NOT pgrep. I've been doing UNIX for well over a decade and never heard of pgrep, although it is installed on my home server apparently. :)
> There are also enhancements like the use of  'cp -l'.  This could be accomplished with the 'ln' command instead on older UNIX versions.  Since it's also used recursively in snapshooter, which 'ln' doesn't support, it makes the command more complex.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.