You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Shai Erera <se...@gmail.com> on 2013/04/22 07:18:10 UTC
nightly-smoke on Cygwin
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to
D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to
D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
* [exec] JAVA7_HOME is D:\d\java\sun-java-70*
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a
different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes
"The literal value for the environment variable." while 'file' is "The
value for the environment variable. *Will be replaced by the absolute
filename of the file by Ant*.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to
D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to
D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File
"D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
1357, in <module>
[exec] main()
[exec] File
"D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File
"D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory:
'/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be
this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full
path. But I wonder if Ant gets confused ... I'm running now with file=
replaced by value=, but it will take some time for this to complete (the
previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
You're right ... I did that b/c the RC validation took 2 hours on my
laptop. While chatting about it with Mike he suggested I try nightly-smoke
to take the RC download out of the picture. And since then I've been
running into those issues.
It's not important at all to me. I thought it's a simple fix to build.xml,
but seeing that it's not, I'll just give it up.
Thanks for your responses.
Shai
On Mon, Apr 22, 2013 at 11:43 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Why do you want to run nightly-smoke? That’s a task intended to be used by
> Jenkins J****
>
> ** **
>
> To test the release candidate, you have to invoke python manually –
> because nightly-smoke only runs the smoketester for the current checkout.*
> ***
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 10:38 AM
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ** **
>
> I'll try the Win32 python build, but if that fails, I'll just not run
> nightly-smoke :).
> Not invoking Ant from Cygwin bash is not an option for me, sorry.****
>
> Shai****
>
> ** **
>
> On Mon, Apr 22, 2013 at 11:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:**
> **
>
> It’s actually worse in your case, as I think you invoke ANT from cygwin’s
> bash:****
>
> Cygwin (bash shell) -calls-> windows (ANT) -class-> Cygwin (python)
> -calls-> windows (Java)****
>
> ****
>
> This makes path conversions horrible and error-prone J the paths need to
> be converted several times. CYGWIN helps with that, but only with
> Environment variables it knows about (and JAVA_* is not one of them). It
> converts PATH and other environment variables, but not those ones. The
> script has to do this.****
>
> ****
>
> My recommendation: Use Win32 python build and don’t use Cygwin’s bash when
> invoking ANT.****
>
> ****
>
> Uwe****
>
> ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Uwe Schindler [mailto:uwe@thetaphi.de]
> *Sent:* Monday, April 22, 2013 8:55 AM
> *To:* dev@lucene.apache.org
> *Subject:* RE: nightly-smoke on Cygwin****
>
> ****
>
> That‘ what the script should do: it must invoke „cygpath“ to resolve
> windows paths and convert to Cygwin paths. There are already code paths in
> it.****
>
> ****
>
> I would still recommend to install a real windows python like Jenkins. The
> main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java)
> and that is likely to fail and is not really a supported configuration. For
> me, with native win32 python downloaded as windows installer all works as
> it should (I did not wait for it to finish).****
>
> ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com <se...@gmail.com>]
> *Sent:* Monday, April 22, 2013 8:48 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like
> I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set
> to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.****
>
> I don't know if the problem is in Ant or the python script. On one hand,
> since the rest of the targets work when I invoke them from Cygwin, I think
> it's the python script. But on the other hand, when I invoke
> smokeTestRelease to verify an RC, and I give it a relative path to place
> the downloaded artifacts, it resolves them properly.****
>
> I think the problem here is that Ant resolves the path to D:\, which
> breaks python. I.e. perhaps if the script replaced D:\ and C:\ with
> /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in
> python? I don't mind giving it a try.
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a
> later stage, because when running parts of the ANT build, the CWD is
> different, so JAVA_HOME passed to ANT is wrong. This is the reason why the
> sysprop is “transformed” to be absolute.****
>
> ****
>
> The Windows-style full path in JAVA7_HOME is correct from the java side
> (it expects a Windows path), so you have to pass a Windows path, not a
> Cygwin path. The python script should convert JAVA7_HOME to its own
> filesystem conventions with the “cygpath” tool when doing checks – and that
> is the bug here.****
>
> ****
>
> The second error is related to you environment args. Whn using –D
> properties passed to ANT, you have to use windows-style paths. The double
> backslash is broken. Mabye use a forward slash if you are invoking from
> Cygwin shell, to prevent the special meaning of \ in cygwin.****
>
> ****
>
> Uwe****
>
> ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 8:18 AM****
>
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Not sure I follow all your arguments. Other targets run fine from Cygwin,
> even while the paths are defined absolute / relative.****
>
> The weird error I got the second time seems unrelated to Cygwin at all
> (nor Ant):
>
>
> /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
> ****
>
> Looks like the python script assumes the argument given to it is already
> relative? I.e. in Java File it would be the equivalent of new File(dir,
> arg)?****
>
> And as for the first replacement of file= by value=, according to the
> <exec> task documentation, I think it's ok to change to value=, because
> we're passing an environment variable, not a file location. Why do other
> targets run fine when they just refer to ${JAVA7_HOME}, but you think we
> must pass a full path reference to the environment variable given to <exec>?
> ****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> Hi,****
>
> ****
>
> Hi this is a general problem with horrible CYGWIN J****
>
> I strongly disagree with replacing file=”…” by value=”..” because this
> makes the general build less portable between platforms and not depend on
> relative directories set correct (especially inside the lucene subfolders,
> where some build.xml set properties for other build.xmls, so all paths
> should be absolute. The problem here is on cygwin’s side: It simply uses
> incompatible path names.****
>
> ****
>
> In my opinion, Java + Cygwin should not be used together for several
> reasons:****
>
> - Java is not a with Cygwin app, so it expects JAVA_HOME as
> full **windows** path – there is also no Cygwin-java available (thanks, J
> )****
>
> - Your python version is expecting /cygdrive paths with forward
> slashes. ANT is not able to produce them unless you use a special Cygwin
> ANT version.****
>
> ****
>
> The fix here is to make the python script check for Cygwin environment and
> fix those path using the dos2unix tool. It already does this, so maybe
> something is missing.****
>
> ****
>
> The more correct variant is not to use Cygwin at all: The Jenkins server
> and I use a native win32 python. If you want a real POSIX/UNIX environment,
> I would use the official “Microsoft” one: Services for Unix, but also not
> in combination with Java. But I never tried the smoke tester with it (I
> don’t use the smoke tester, because I prefer to check the artifacts
> manually before giving my vote. The +1 is coming from me, not a script).**
> **
>
> ****
>
> Uwe****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 7:50 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Changing <arg file= to <arg value= didn't help, I still got the same
> error.****
>
>
> I should note that I am able to run smokeTestRelease fine, when validating
> RCs.****
>
> ****
>
> Any ideas (besides moving to Linux :))?****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".****
>
> Anyone can confirm it is ok to make this change?****
>
> After making this change, I hit another error:****
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8****
>
>
> [exec] Traceback (most recent call last):****
>
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
> ****
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:****
>
>
> <arg file="${fakeReleaseTmp}"/>****
>
> ****
>
> The property fakeReleaseTmp is defined as relative path:****
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
> ****
>
> ****
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).****
>
> Again, can anyone confirm this change is safe?
>
> Shai****
>
> ****
>
> ****
>
> ****
>
> ** **
>
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
Why do you want to run nightly-smoke? That’s a task intended to be used by Jenkins J
To test the release candidate, you have to invoke python manually – because nightly-smoke only runs the smoketester for the current checkout.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 10:38 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
I'll try the Win32 python build, but if that fails, I'll just not run nightly-smoke :).
Not invoking Ant from Cygwin bash is not an option for me, sorry.
Shai
On Mon, Apr 22, 2013 at 11:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
It’s actually worse in your case, as I think you invoke ANT from cygwin’s bash:
Cygwin (bash shell) -calls-> windows (ANT) -class-> Cygwin (python) -calls-> windows (Java)
This makes path conversions horrible and error-prone J the paths need to be converted several times. CYGWIN helps with that, but only with Environment variables it knows about (and JAVA_* is not one of them). It converts PATH and other environment variables, but not those ones. The script has to do this.
My recommendation: Use Win32 python build and don’t use Cygwin’s bash when invoking ANT.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, April 22, 2013 8:55 AM
To: dev@lucene.apache.org
Subject: RE: nightly-smoke on Cygwin
That‘ what the script should do: it must invoke „cygpath“ to resolve windows paths and convert to Cygwin paths. There are already code paths in it.
I would still recommend to install a real windows python like Jenkins. The main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java) and that is likely to fail and is not really a supported configuration. For me, with native win32 python downloaded as windows installer all works as it should (I did not wait for it to finish).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:48 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.
I don't know if the problem is in Ant or the python script. On one hand, since the rest of the targets work when I invoke them from Cygwin, I think it's the python script. But on the other hand, when I invoke smokeTestRelease to verify an RC, and I give it a relative path to place the downloaded artifacts, it resolves them properly.
I think the problem here is that Ant resolves the path to D:\, which breaks python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in python? I don't mind giving it a try.
Shai
On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute.
The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here.
The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:18 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)?
And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
I'll try the Win32 python build, but if that fails, I'll just not run
nightly-smoke :).
Not invoking Ant from Cygwin bash is not an option for me, sorry.
Shai
On Mon, Apr 22, 2013 at 11:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> It’s actually worse in your case, as I think you invoke ANT from cygwin’s
> bash:****
>
> Cygwin (bash shell) -calls-> windows (ANT) -class-> Cygwin (python)
> -calls-> windows (Java)****
>
> ** **
>
> This makes path conversions horrible and error-prone J the paths need to
> be converted several times. CYGWIN helps with that, but only with
> Environment variables it knows about (and JAVA_* is not one of them). It
> converts PATH and other environment variables, but not those ones. The
> script has to do this.****
>
> ** **
>
> My recommendation: Use Win32 python build and don’t use Cygwin’s bash when
> invoking ANT.****
>
> ** **
>
> Uwe****
>
> ** **
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Uwe Schindler [mailto:uwe@thetaphi.de]
> *Sent:* Monday, April 22, 2013 8:55 AM
> *To:* dev@lucene.apache.org
> *Subject:* RE: nightly-smoke on Cygwin****
>
> ** **
>
> That‘ what the script should do: it must invoke „cygpath“ to resolve
> windows paths and convert to Cygwin paths. There are already code paths in
> it.****
>
> ** **
>
> I would still recommend to install a real windows python like Jenkins. The
> main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java)
> and that is likely to fail and is not really a supported configuration. For
> me, with native win32 python downloaded as windows installer all works as
> it should (I did not wait for it to finish).****
>
> ** **
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Shai Erera [mailto:serera@gmail.com <se...@gmail.com>]
> *Sent:* Monday, April 22, 2013 8:48 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ** **
>
> I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like
> I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set
> to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.****
>
> I don't know if the problem is in Ant or the python script. On one hand,
> since the rest of the targets work when I invoke them from Cygwin, I think
> it's the python script. But on the other hand, when I invoke
> smokeTestRelease to verify an RC, and I give it a relative path to place
> the downloaded artifacts, it resolves them properly.****
>
> I think the problem here is that Ant resolves the path to D:\, which
> breaks python. I.e. perhaps if the script replaced D:\ and C:\ with
> /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in
> python? I don't mind giving it a try.
>
> Shai****
>
> ** **
>
> On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a
> later stage, because when running parts of the ANT build, the CWD is
> different, so JAVA_HOME passed to ANT is wrong. This is the reason why the
> sysprop is “transformed” to be absolute.****
>
> ****
>
> The Windows-style full path in JAVA7_HOME is correct from the java side
> (it expects a Windows path), so you have to pass a Windows path, not a
> Cygwin path. The python script should convert JAVA7_HOME to its own
> filesystem conventions with the “cygpath” tool when doing checks – and that
> is the bug here.****
>
> ****
>
> The second error is related to you environment args. Whn using –D
> properties passed to ANT, you have to use windows-style paths. The double
> backslash is broken. Mabye use a forward slash if you are invoking from
> Cygwin shell, to prevent the special meaning of \ in cygwin.****
>
> ****
>
> Uwe****
>
> ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 8:18 AM****
>
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Not sure I follow all your arguments. Other targets run fine from Cygwin,
> even while the paths are defined absolute / relative.****
>
> The weird error I got the second time seems unrelated to Cygwin at all
> (nor Ant):
>
>
> /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
> ****
>
> Looks like the python script assumes the argument given to it is already
> relative? I.e. in Java File it would be the equivalent of new File(dir,
> arg)?****
>
> And as for the first replacement of file= by value=, according to the
> <exec> task documentation, I think it's ok to change to value=, because
> we're passing an environment variable, not a file location. Why do other
> targets run fine when they just refer to ${JAVA7_HOME}, but you think we
> must pass a full path reference to the environment variable given to <exec>?
> ****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> Hi,****
>
> ****
>
> Hi this is a general problem with horrible CYGWIN J****
>
> I strongly disagree with replacing file=”…” by value=”..” because this
> makes the general build less portable between platforms and not depend on
> relative directories set correct (especially inside the lucene subfolders,
> where some build.xml set properties for other build.xmls, so all paths
> should be absolute. The problem here is on cygwin’s side: It simply uses
> incompatible path names.****
>
> ****
>
> In my opinion, Java + Cygwin should not be used together for several
> reasons:****
>
> - Java is not a with Cygwin app, so it expects JAVA_HOME as
> full **windows** path – there is also no Cygwin-java available (thanks, J
> )****
>
> - Your python version is expecting /cygdrive paths with forward
> slashes. ANT is not able to produce them unless you use a special Cygwin
> ANT version.****
>
> ****
>
> The fix here is to make the python script check for Cygwin environment and
> fix those path using the dos2unix tool. It already does this, so maybe
> something is missing.****
>
> ****
>
> The more correct variant is not to use Cygwin at all: The Jenkins server
> and I use a native win32 python. If you want a real POSIX/UNIX environment,
> I would use the official “Microsoft” one: Services for Unix, but also not
> in combination with Java. But I never tried the smoke tester with it (I
> don’t use the smoke tester, because I prefer to check the artifacts
> manually before giving my vote. The +1 is coming from me, not a script).**
> **
>
> ****
>
> Uwe****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 7:50 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Changing <arg file= to <arg value= didn't help, I still got the same
> error.****
>
>
> I should note that I am able to run smokeTestRelease fine, when validating
> RCs.****
>
> ****
>
> Any ideas (besides moving to Linux :))?****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".****
>
> Anyone can confirm it is ok to make this change?****
>
> After making this change, I hit another error:****
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8****
>
>
> [exec] Traceback (most recent call last):****
>
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
> ****
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:****
>
>
> <arg file="${fakeReleaseTmp}"/>****
>
> ****
>
> The property fakeReleaseTmp is defined as relative path:****
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
> ****
>
> ****
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).****
>
> Again, can anyone confirm this change is safe?
>
> Shai****
>
> ****
>
> ****
>
> ** **
>
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
It’s actually worse in your case, as I think you invoke ANT from cygwin’s bash:
Cygwin (bash shell) -calls-> windows (ANT) -class-> Cygwin (python) -calls-> windows (Java)
This makes path conversions horrible and error-prone J the paths need to be converted several times. CYGWIN helps with that, but only with Environment variables it knows about (and JAVA_* is not one of them). It converts PATH and other environment variables, but not those ones. The script has to do this.
My recommendation: Use Win32 python build and don’t use Cygwin’s bash when invoking ANT.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, April 22, 2013 8:55 AM
To: dev@lucene.apache.org
Subject: RE: nightly-smoke on Cygwin
That‘ what the script should do: it must invoke „cygpath“ to resolve windows paths and convert to Cygwin paths. There are already code paths in it.
I would still recommend to install a real windows python like Jenkins. The main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java) and that is likely to fail and is not really a supported configuration. For me, with native win32 python downloaded as windows installer all works as it should (I did not wait for it to finish).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:48 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.
I don't know if the problem is in Ant or the python script. On one hand, since the rest of the targets work when I invoke them from Cygwin, I think it's the python script. But on the other hand, when I invoke smokeTestRelease to verify an RC, and I give it a relative path to place the downloaded artifacts, it resolves them properly.
I think the problem here is that Ant resolves the path to D:\, which breaks python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in python? I don't mind giving it a try.
Shai
On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute.
The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here.
The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:18 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)?
And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
According to build.xml, you have to pass –Dpython32.exe=…. To ant, so it needs Python 3.x. Theoretically 3.3 should work, just try it. On Jenkins we have the officially supported versions installed. Refer to the beginning of http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/lastCompletedBuild/consoleFull for the full cmd line arguments to make ANT use the correct python:
[Lucene-Solr-trunk-Windows] $ cmd.exe /C '"C:\Users\JenkinsSlave\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2\bin\ant.bat -Dpython32.exe=C:\Python32\python.exe -Dperl.exe=C:\Perl\bin\perl.exe "-Dargs=-server -XX:+UseSerialGC" -Dtests.jvms=1 -Dpython.exe=C:\Python27\python.exe jenkins-hourly && exit %%ERRORLEVEL%%"'
I am not sure if python 2.7 is still needed for such a build, I have to find out, in the past some scripts were in python 2 format.
If I have time, I will setup another windows build with smoketester in windows.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 9:30 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Will it work with Python 3.3?
Shai
On Mon, Apr 22, 2013 at 9:54 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
That‘ what the script should do: it must invoke „cygpath“ to resolve windows paths and convert to Cygwin paths. There are already code paths in it.
I would still recommend to install a real windows python like Jenkins. The main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java) and that is likely to fail and is not really a supported configuration. For me, with native win32 python downloaded as windows installer all works as it should (I did not wait for it to finish).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:48 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.
I don't know if the problem is in Ant or the python script. On one hand, since the rest of the targets work when I invoke them from Cygwin, I think it's the python script. But on the other hand, when I invoke smokeTestRelease to verify an RC, and I give it a relative path to place the downloaded artifacts, it resolves them properly.
I think the problem here is that Ant resolves the path to D:\, which breaks python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in python? I don't mind giving it a try.
Shai
On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute.
The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here.
The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:18 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)?
And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
Will it work with Python 3.3?
Shai
On Mon, Apr 22, 2013 at 9:54 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> That‘ what the script should do: it must invoke „cygpath“ to resolve
> windows paths and convert to Cygwin paths. There are already code paths in
> it.****
>
> ** **
>
> I would still recommend to install a real windows python like Jenkins. The
> main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java)
> and that is likely to fail and is not really a supported configuration. For
> me, with native win32 python downloaded as windows installer all works as
> it should (I did not wait for it to finish).****
>
> ** **
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 8:48 AM
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ** **
>
> I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like
> I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set
> to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.****
>
> I don't know if the problem is in Ant or the python script. On one hand,
> since the rest of the targets work when I invoke them from Cygwin, I think
> it's the python script. But on the other hand, when I invoke
> smokeTestRelease to verify an RC, and I give it a relative path to place
> the downloaded artifacts, it resolves them properly.****
>
> I think the problem here is that Ant resolves the path to D:\, which
> breaks python. I.e. perhaps if the script replaced D:\ and C:\ with
> /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in
> python? I don't mind giving it a try.
>
> Shai****
>
> ** **
>
> On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a
> later stage, because when running parts of the ANT build, the CWD is
> different, so JAVA_HOME passed to ANT is wrong. This is the reason why the
> sysprop is “transformed” to be absolute.****
>
> ****
>
> The Windows-style full path in JAVA7_HOME is correct from the java side
> (it expects a Windows path), so you have to pass a Windows path, not a
> Cygwin path. The python script should convert JAVA7_HOME to its own
> filesystem conventions with the “cygpath” tool when doing checks – and that
> is the bug here.****
>
> ****
>
> The second error is related to you environment args. Whn using –D
> properties passed to ANT, you have to use windows-style paths. The double
> backslash is broken. Mabye use a forward slash if you are invoking from
> Cygwin shell, to prevent the special meaning of \ in cygwin.****
>
> ****
>
> Uwe****
>
> ****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 8:18 AM****
>
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Not sure I follow all your arguments. Other targets run fine from Cygwin,
> even while the paths are defined absolute / relative.****
>
> The weird error I got the second time seems unrelated to Cygwin at all
> (nor Ant):
>
>
> /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
> ****
>
> Looks like the python script assumes the argument given to it is already
> relative? I.e. in Java File it would be the equivalent of new File(dir,
> arg)?****
>
> And as for the first replacement of file= by value=, according to the
> <exec> task documentation, I think it's ok to change to value=, because
> we're passing an environment variable, not a file location. Why do other
> targets run fine when they just refer to ${JAVA7_HOME}, but you think we
> must pass a full path reference to the environment variable given to <exec>?
> ****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> Hi,****
>
> ****
>
> Hi this is a general problem with horrible CYGWIN J****
>
> I strongly disagree with replacing file=”…” by value=”..” because this
> makes the general build less portable between platforms and not depend on
> relative directories set correct (especially inside the lucene subfolders,
> where some build.xml set properties for other build.xmls, so all paths
> should be absolute. The problem here is on cygwin’s side: It simply uses
> incompatible path names.****
>
> ****
>
> In my opinion, Java + Cygwin should not be used together for several
> reasons:****
>
> - Java is not a with Cygwin app, so it expects JAVA_HOME as
> full **windows** path – there is also no Cygwin-java available (thanks, J
> )****
>
> - Your python version is expecting /cygdrive paths with forward
> slashes. ANT is not able to produce them unless you use a special Cygwin
> ANT version.****
>
> ****
>
> The fix here is to make the python script check for Cygwin environment and
> fix those path using the dos2unix tool. It already does this, so maybe
> something is missing.****
>
> ****
>
> The more correct variant is not to use Cygwin at all: The Jenkins server
> and I use a native win32 python. If you want a real POSIX/UNIX environment,
> I would use the official “Microsoft” one: Services for Unix, but also not
> in combination with Java. But I never tried the smoke tester with it (I
> don’t use the smoke tester, because I prefer to check the artifacts
> manually before giving my vote. The +1 is coming from me, not a script).**
> **
>
> ****
>
> Uwe****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 7:50 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Changing <arg file= to <arg value= didn't help, I still got the same
> error.****
>
>
> I should note that I am able to run smokeTestRelease fine, when validating
> RCs.****
>
> ****
>
> Any ideas (besides moving to Linux :))?****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".****
>
> Anyone can confirm it is ok to make this change?****
>
> After making this change, I hit another error:****
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8****
>
>
> [exec] Traceback (most recent call last):****
>
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
> ****
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:****
>
>
> <arg file="${fakeReleaseTmp}"/>****
>
> ****
>
> The property fakeReleaseTmp is defined as relative path:****
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
> ****
>
> ****
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).****
>
> Again, can anyone confirm this change is safe?
>
> Shai****
>
> ****
>
> ****
>
> ** **
>
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
That‘ what the script should do: it must invoke „cygpath“ to resolve windows paths and convert to Cygwin paths. There are already code paths in it.
I would still recommend to install a real windows python like Jenkins. The main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java) and that is likely to fail and is not really a supported configuration. For me, with native win32 python downloaded as windows installer all works as it should (I did not wait for it to finish).
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:48 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.
I don't know if the problem is in Ant or the python script. On one hand, since the rest of the targets work when I invoke them from Cygwin, I think it's the python script. But on the other hand, when I invoke smokeTestRelease to verify an RC, and I give it a relative path to place the downloaded artifacts, it resolves them properly.
I think the problem here is that Ant resolves the path to D:\, which breaks python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in python? I don't mind giving it a try.
Shai
On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute.
The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here.
The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:18 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)?
And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like
I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set
to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7.
I don't know if the problem is in Ant or the python script. On one hand,
since the rest of the targets work when I invoke them from Cygwin, I think
it's the python script. But on the other hand, when I invoke
smokeTestRelease to verify an RC, and I give it a relative path to place
the downloaded artifacts, it resolves them properly.
I think the problem here is that Ant resolves the path to D:\, which breaks
python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d
and /cygdrive/c it will work? Do you know how to do that in python? I don't
mind giving it a try.
Shai
On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a
> later stage, because when running parts of the ANT build, the CWD is
> different, so JAVA_HOME passed to ANT is wrong. This is the reason why the
> sysprop is “transformed” to be absolute.****
>
> ** **
>
> The Windows-style full path in JAVA7_HOME is correct from the java side
> (it expects a Windows path), so you have to pass a Windows path, not a
> Cygwin path. The python script should convert JAVA7_HOME to its own
> filesystem conventions with the “cygpath” tool when doing checks – and that
> is the bug here.****
>
> ** **
>
> The second error is related to you environment args. Whn using –D
> properties passed to ANT, you have to use windows-style paths. The double
> backslash is broken. Mabye use a forward slash if you are invoking from
> Cygwin shell, to prevent the special meaning of \ in cygwin.****
>
> ** **
>
> Uwe****
>
> ** **
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 8:18 AM
>
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ** **
>
> Not sure I follow all your arguments. Other targets run fine from Cygwin,
> even while the paths are defined absolute / relative.****
>
> The weird error I got the second time seems unrelated to Cygwin at all
> (nor Ant):
>
>
> /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
> ****
>
> Looks like the python script assumes the argument given to it is already
> relative? I.e. in Java File it would be the equivalent of new File(dir,
> arg)?****
>
> And as for the first replacement of file= by value=, according to the
> <exec> task documentation, I think it's ok to change to value=, because
> we're passing an environment variable, not a file location. Why do other
> targets run fine when they just refer to ${JAVA7_HOME}, but you think we
> must pass a full path reference to the environment variable given to <exec>?
> ****
>
> Shai****
>
> ** **
>
> On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:***
> *
>
> Hi,****
>
> ****
>
> Hi this is a general problem with horrible CYGWIN J****
>
> I strongly disagree with replacing file=”…” by value=”..” because this
> makes the general build less portable between platforms and not depend on
> relative directories set correct (especially inside the lucene subfolders,
> where some build.xml set properties for other build.xmls, so all paths
> should be absolute. The problem here is on cygwin’s side: It simply uses
> incompatible path names.****
>
> ****
>
> In my opinion, Java + Cygwin should not be used together for several
> reasons:****
>
> - Java is not a with Cygwin app, so it expects JAVA_HOME as
> full **windows** path – there is also no Cygwin-java available (thanks, J
> )****
>
> - Your python version is expecting /cygdrive paths with forward
> slashes. ANT is not able to produce them unless you use a special Cygwin
> ANT version.****
>
> ****
>
> The fix here is to make the python script check for Cygwin environment and
> fix those path using the dos2unix tool. It already does this, so maybe
> something is missing.****
>
> ****
>
> The more correct variant is not to use Cygwin at all: The Jenkins server
> and I use a native win32 python. If you want a real POSIX/UNIX environment,
> I would use the official “Microsoft” one: Services for Unix, but also not
> in combination with Java. But I never tried the smoke tester with it (I
> don’t use the smoke tester, because I prefer to check the artifacts
> manually before giving my vote. The +1 is coming from me, not a script).**
> **
>
> ****
>
> Uwe****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ****
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 7:50 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ****
>
> Changing <arg file= to <arg value= didn't help, I still got the same
> error.****
>
>
> I should note that I am able to run smokeTestRelease fine, when validating
> RCs.****
>
> ****
>
> Any ideas (besides moving to Linux :))?****
>
> Shai****
>
> ****
>
> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".****
>
> Anyone can confirm it is ok to make this change?****
>
> After making this change, I hit another error:****
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8****
>
>
> [exec] Traceback (most recent call last):****
>
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
> ****
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:****
>
>
> <arg file="${fakeReleaseTmp}"/>****
>
> ****
>
> The property fakeReleaseTmp is defined as relative path:****
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
> ****
>
> ****
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).****
>
> Again, can anyone confirm this change is safe?
>
> Shai****
>
> ****
>
> ** **
>
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute.
The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here.
The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 8:18 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)?
And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
I checked smokeTestRelease.py (line 1298) and this is how tmpDir is
initialized:
tmpDir = os.path.abspath(sys.argv[4])
So indeed seems that perhaps this can be resolved in the script, because
what's passed to the script is d:\dev..., which I assume os.path.abspath
does not recognize as absolute path and therefore adds the CWD to it. But
I'm just guessing... I have no idea how to fix smokeTestRelease.py though.
Shai
On Mon, Apr 22, 2013 at 9:17 AM, Shai Erera <se...@gmail.com> wrote:
> Not sure I follow all your arguments. Other targets run fine from Cygwin,
> even while the paths are defined absolute / relative.
>
> The weird error I got the second time seems unrelated to Cygwin at all
> (nor Ant):
>
>
>
> /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
>
> Looks like the python script assumes the argument given to it is already
> relative? I.e. in Java File it would be the equivalent of new File(dir,
> arg)?
>
> And as for the first replacement of file= by value=, according to the
> <exec> task documentation, I think it's ok to change to value=, because
> we're passing an environment variable, not a file location. Why do other
> targets run fine when they just refer to ${JAVA7_HOME}, but you think we
> must pass a full path reference to the environment variable given to <exec>?
>
> Shai
>
>
> On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Hi,****
>>
>> ** **
>>
>> Hi this is a general problem with horrible CYGWIN J****
>>
>> I strongly disagree with replacing file=”…” by value=”..” because this
>> makes the general build less portable between platforms and not depend on
>> relative directories set correct (especially inside the lucene subfolders,
>> where some build.xml set properties for other build.xmls, so all paths
>> should be absolute. The problem here is on cygwin’s side: It simply uses
>> incompatible path names.****
>>
>> ** **
>>
>> In my opinion, Java + Cygwin should not be used together for several
>> reasons:****
>>
>> **- **Java is not a with Cygwin app, so it expects JAVA_HOME
>> as full **windows** path – there is also no Cygwin-java available
>> (thanks, J)****
>>
>> **- **Your python version is expecting /cygdrive paths with
>> forward slashes. ANT is not able to produce them unless you use a special
>> Cygwin ANT version.****
>>
>> ** **
>>
>> The fix here is to make the python script check for Cygwin environment
>> and fix those path using the dos2unix tool. It already does this, so maybe
>> something is missing.****
>>
>> ** **
>>
>> The more correct variant is not to use Cygwin at all: The Jenkins server
>> and I use a native win32 python. If you want a real POSIX/UNIX environment,
>> I would use the official “Microsoft” one: Services for Unix, but also not
>> in combination with Java. But I never tried the smoke tester with it (I
>> don’t use the smoke tester, because I prefer to check the artifacts
>> manually before giving my vote. The +1 is coming from me, not a script).*
>> ***
>>
>> ** **
>>
>> Uwe****
>>
>> -----****
>>
>> Uwe Schindler****
>>
>> H.-H.-Meier-Allee 63, D-28213 Bremen****
>>
>> http://www.thetaphi.de****
>>
>> eMail: uwe@thetaphi.de****
>>
>> ** **
>>
>> *From:* Shai Erera [mailto:serera@gmail.com]
>> *Sent:* Monday, April 22, 2013 7:50 AM
>> *To:* dev@lucene.apache.org
>> *Subject:* Re: nightly-smoke on Cygwin****
>>
>> ** **
>>
>> Changing <arg file= to <arg value= didn't help, I still got the same
>> error.****
>>
>>
>> I should note that I am able to run smokeTestRelease fine, when
>> validating RCs.****
>>
>> ** **
>>
>> Any ideas (besides moving to Linux :))?****
>>
>> Shai****
>>
>> ** **
>>
>> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>>
>> Hi
>>
>> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors
>> as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
>> work with other ant targets, e.g. precommit, test etc.
>>
>> prepare-release-no-sign:
>> [mkdir] Created dir:
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
>> [copy] Copying 401 files to
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
>> [copy] Copying 194 files to
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
>> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>>
>> This seems to be caused by this line:
>>
>> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>>
>> If I replace it by this, I don't get the error anymore (but there's a
>> different one, see below):
>>
>> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>>
>> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
>> denotes "The literal value for the environment variable." while 'file' is
>> "The value for the environment variable. *Will be replaced by the
>> absolute filename of the file by Ant*.".****
>>
>> Anyone can confirm it is ok to make this change?****
>>
>> After making this change, I hit another error:****
>>
>> prepare-release-no-sign:
>> [mkdir] Created dir:
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
>> [copy] Copying 401 files to
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
>> [copy] Copying 194 files to
>> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>>
>> [exec] JAVA7_HOME is /d/java/sun-java-70
>> [exec] NOTE: output encoding is UTF-8****
>>
>>
>> [exec] Traceback (most recent call last):****
>>
>> [exec] File
>> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
>> 1357, in <module>
>> [exec] main()
>> [exec] File
>> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
>> 1301, in main
>> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
>> [exec] File
>> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
>> 1312, in smokeTest
>> [exec] os.makedirs(tmpDir)
>> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
>> [exec] mkdir(name, mode)
>> [exec] OSError: [Errno 2] No such file or directory:
>> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
>> ****
>>
>> See the last line, how the fakeRelease location is corrupt. Could it be
>> this line:****
>>
>>
>> <arg file="${fakeReleaseTmp}"/>****
>>
>> ** **
>>
>> The property fakeReleaseTmp is defined as relative path:****
>>
>> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
>> ****
>>
>> ** **
>>
>> In Eclipse, when I hover over the property, it displays the correct full
>> path. But I wonder if Ant gets confused ... I'm running now with file=
>> replaced by value=, but it will take some time for this to complete (the
>> previous error I got was after 48 minutes).****
>>
>> Again, can anyone confirm this change is safe?
>>
>> Shai****
>>
>> ** **
>>
>
>
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
Not sure I follow all your arguments. Other targets run fine from Cygwin,
even while the paths are defined absolute / relative.
The weird error I got the second time seems unrelated to Cygwin at all (nor
Ant):
/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp
Looks like the python script assumes the argument given to it is already
relative? I.e. in Java File it would be the equivalent of new File(dir,
arg)?
And as for the first replacement of file= by value=, according to the
<exec> task documentation, I think it's ok to change to value=, because
we're passing an environment variable, not a file location. Why do other
targets run fine when they just refer to ${JAVA7_HOME}, but you think we
must pass a full path reference to the environment variable given to <exec>?
Shai
On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,****
>
> ** **
>
> Hi this is a general problem with horrible CYGWIN J****
>
> I strongly disagree with replacing file=”…” by value=”..” because this
> makes the general build less portable between platforms and not depend on
> relative directories set correct (especially inside the lucene subfolders,
> where some build.xml set properties for other build.xmls, so all paths
> should be absolute. The problem here is on cygwin’s side: It simply uses
> incompatible path names.****
>
> ** **
>
> In my opinion, Java + Cygwin should not be used together for several
> reasons:****
>
> **- **Java is not a with Cygwin app, so it expects JAVA_HOME as
> full **windows** path – there is also no Cygwin-java available (thanks, J
> )****
>
> **- **Your python version is expecting /cygdrive paths with
> forward slashes. ANT is not able to produce them unless you use a special
> Cygwin ANT version.****
>
> ** **
>
> The fix here is to make the python script check for Cygwin environment and
> fix those path using the dos2unix tool. It already does this, so maybe
> something is missing.****
>
> ** **
>
> The more correct variant is not to use Cygwin at all: The Jenkins server
> and I use a native win32 python. If you want a real POSIX/UNIX environment,
> I would use the official “Microsoft” one: Services for Unix, but also not
> in combination with Java. But I never tried the smoke tester with it (I
> don’t use the smoke tester, because I prefer to check the artifacts
> manually before giving my vote. The +1 is coming from me, not a script).**
> **
>
> ** **
>
> Uwe****
>
> -----****
>
> Uwe Schindler****
>
> H.-H.-Meier-Allee 63, D-28213 Bremen****
>
> http://www.thetaphi.de****
>
> eMail: uwe@thetaphi.de****
>
> ** **
>
> *From:* Shai Erera [mailto:serera@gmail.com]
> *Sent:* Monday, April 22, 2013 7:50 AM
> *To:* dev@lucene.apache.org
> *Subject:* Re: nightly-smoke on Cygwin****
>
> ** **
>
> Changing <arg file= to <arg value= didn't help, I still got the same
> error.****
>
>
> I should note that I am able to run smokeTestRelease fine, when validating
> RCs.****
>
> ** **
>
> Any ideas (besides moving to Linux :))?****
>
> Shai****
>
> ** **
>
> On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:****
>
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*****
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>****
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>****
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".****
>
> Anyone can confirm it is ok to make this change?****
>
> After making this change, I hit another error:****
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr****
>
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8****
>
>
> [exec] Traceback (most recent call last):****
>
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
> ****
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:****
>
>
> <arg file="${fakeReleaseTmp}"/>****
>
> ** **
>
> The property fakeReleaseTmp is defined as relative path:****
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
> ****
>
> ** **
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).****
>
> Again, can anyone confirm this change is safe?
>
> Shai****
>
> ** **
>
RE: nightly-smoke on Cygwin
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Hi this is a general problem with horrible CYGWIN J
I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names.
In my opinion, Java + Cygwin should not be used together for several reasons:
- Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J)
- Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version.
The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing.
The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
<http://www.thetaphi.de/> http://www.thetaphi.de
eMail: uwe@thetaphi.de
From: Shai Erera [mailto:serera@gmail.com]
Sent: Monday, April 22, 2013 7:50 AM
To: dev@lucene.apache.org
Subject: Re: nightly-smoke on Cygwin
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
Hi
I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc.
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is D:\d\java\sun-java-70
This seems to be caused by this line:
<env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
If I replace it by this, I don't get the error anymore (but there's a different one, see below):
<env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.".
Anyone can confirm it is ok to make this change?
After making this change, I hit another error:
prepare-release-no-sign:
[mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
[copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
[copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
[exec] JAVA7_HOME is /d/java/sun-java-70
[exec] NOTE: output encoding is UTF-8
[exec] Traceback (most recent call last):
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module>
[exec] main()
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main
[exec] smokeTest(baseURL, version, tmpDir, isSigned)
[exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest
[exec] os.makedirs(tmpDir)
[exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
[exec] mkdir(name, mode)
[exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
See the last line, how the fakeRelease location is corrupt. Could it be this line:
<arg file="${fakeReleaseTmp}"/>
The property fakeReleaseTmp is defined as relative path:
<property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes).
Again, can anyone confirm this change is safe?
Shai
Re: nightly-smoke on Cygwin
Posted by Shai Erera <se...@gmail.com>.
Changing <arg file= to <arg value= didn't help, I still got the same error.
I should note that I am able to run smokeTestRelease fine, when validating
RCs.
Any ideas (besides moving to Linux :))?
Shai
On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <se...@gmail.com> wrote:
> Hi
>
> I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as
> depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to
> work with other ant targets, e.g. precommit, test etc.
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> * [exec] JAVA7_HOME is D:\d\java\sun-java-70*
>
> This seems to be caused by this line:
>
> <env key="JAVA7_HOME" file="${JAVA7_HOME}"/>
>
> If I replace it by this, I don't get the error anymore (but there's a
> different one, see below):
>
> <env key="JAVA7_HOME" value="${JAVA7_HOME}"/>
>
> According to http://ant.apache.org/manual/Tasks/exec.html, 'value'
> denotes "The literal value for the environment variable." while 'file' is
> "The value for the environment variable. *Will be replaced by the
> absolute filename of the file by Ant*.".
>
> Anyone can confirm it is ok to make this change?
>
> After making this change, I hit another error:
>
> prepare-release-no-sign:
> [mkdir] Created dir:
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease
> [copy] Copying 401 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene
> [copy] Copying 194 files to
> D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr
> [exec] JAVA7_HOME is /d/java/sun-java-70
> [exec] NOTE: output encoding is UTF-8
>
> [exec] Traceback (most recent call last):
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1357, in <module>
> [exec] main()
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1301, in main
> [exec] smokeTest(baseURL, version, tmpDir, isSigned)
> [exec] File
> "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line
> 1312, in smokeTest
> [exec] os.makedirs(tmpDir)
> [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs
> [exec] mkdir(name, mode)
> [exec] OSError: [Errno 2] No such file or directory:
> '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp'
>
> See the last line, how the fakeRelease location is corrupt. Could it be
> this line:
>
> <arg file="${fakeReleaseTmp}"/>
>
> The property fakeReleaseTmp is defined as relative path:
>
> <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/>
>
> In Eclipse, when I hover over the property, it displays the correct full
> path. But I wonder if Ant gets confused ... I'm running now with file=
> replaced by value=, but it will take some time for this to complete (the
> previous error I got was after 48 minutes).
>
> Again, can anyone confirm this change is safe?
>
> Shai
>