Re: Disk space requirement for building on Windows

[Andy Seaborne <an...@apache.org>]

Changing the tests to make them run isn't testing any more.

We have been through this issues many times now.  It's a nuisance, but 
we have Jenkins, though the Windows slaves are less stable and we 
sometimes have to disable the job.

Helping out with the Windows slaves is a *much* better solution.

I reviewed what had been run recently enough, together with noting 
changes in TDB (none relevant) when I was doing the build to be happy 
that testing had been done at some point.

If someone has a better solution - great.  It has to be a proper 
solution though and, as reading the Java bug reports shows, "works for 
me at the moment" does not reliably mean "works", for everyone or in the 
future.

There are several different JVMs.

	Andy

On 10/03/15 11:14, Stian Soiland-Reyes wrote:
> The System.gc() is not guaranteed to work - but it did work for me.
> That is what I meant by a 'best effort' within the test. The test does
> not require all those old test folders to hang around.
>
> Could we do a workaround for the test to set say a tinier block-size
> instead of 8 MB?
>
>
> As I have been unable to build all of Jena on Windows - what is the
> actual disk space requirement? It should at least be documented in the
> README.
>
> On 10 March 2015 at 11:07, Rob Vesse <rv...@dotnetrdf.org> wrote:
>> Using an alternative approach would not make any difference
>>
>> It is a fundamental bug in Windows memory mapped files that means that a
>> JVM can never guarantee to completely release memory mapped files while
>> the JVM is alive.
>>
>> Andy has posted this many times on threads about TDB on Windows in the
>> past.  No workaround we could attempt could ever solve the issue on
>> Windows so there is really no point in expending effort changing something
>> low level that otherwise works fine across multiple platforms.
>>
>> Rob
>>
>> On 10/03/2015 10:25, "Stian Soiland-Reyes" <st...@apache.org> wrote:
>>
>>> Thanks, Rob!
>>>
>>> I tried looking yesterday at ways to reduce the disk space
>>> requirements when building on Windows - including truncating the files
>>> after closing. This seems to require deep changes into TDBs
>>> ChannelManager which keeps the corresponding FileChannels - perhaps a
>>> new method for that purpose?
>>>
>>> https://github.com/apache/jena/blob/master/jena-tdb/src/main/java/com/hp/h
>>> pl/jena/tdb/base/file/ChannelManager.java
>>>
>>>
>>> It seems on Windows with Oracle/OpenJDK you can call System.gc() to
>>> (hopefully) release the ByteBuffers that lock the memory regions (and
>>> then making the files deletable) - but this adds a significant
>>> overhead. The dispose methods on ByteBufferImpls are not easily
>>> accessible - you would need some introspection hackery to get hold of
>>> that cleaner() and that would of course only work on Oracle/OpenJDK.
>>> as fc.map() still does the same thing.
>>>
>>> Close your eyes -  GPL3!
>>> https://github.com/stain/jdk8u/blob/master/src/share/classes/java/nio/Dire
>>> ct-X-Buffer.java.template#L72
>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b
>>> 132/java/nio/DirectByteBuffer.java/#72
>>>
>>>
>>> I tried using the FileChannels from JDK7 NIO2 (e.g.
>>> FileChannel.open(Path)) instead of through RandomAccessFile - but it
>>> did not make any difference
>>>
>>>
>>> Perhaps System.gc() is not worth it in general (* on Windows) when
>>> closing a dataset - I tried to modify the ChannelManager to always do
>>> this on release, it meant each test in jena-jdbc-tdb took 1.5s instead
>>> of 0.2s, but it did allow me to delete the used folders from target/
>>> while the JVM/test was running.
>>>
>>> For the tests we could do something like for every 10 tests do
>>> System.gc() and wipe the old data.
>>>
>>> Perhaps Fuseki 2 could do System.gc() on [Remove] && SystemTDB.isWindows.
>>>
>>>
>>>
>>> On 10 March 2015 at 10:00, ASF GitHub Bot (JIRA) <ji...@apache.org> wrote:
>>>>
>>>>      [
>>>> https://issues.apache.org/jira/browse/JENA-897?page=com.atlassian.jira.pl
>>>> ugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354617#com
>>>> ment-14354617 ]
>>>>
>>>> ASF GitHub Bot commented on JENA-897:
>>>> -------------------------------------
>>>>
>>>> Github user asfgit closed the pull request at:
>>>>
>>>>      https://github.com/apache/jena/pull/41
>>>>
>>>>
>>>>> jena-jdbc-tdb tests use %TEMP% instead of target/
>>>>> -------------------------------------------------
>>>>>
>>>>>                  Key: JENA-897
>>>>>                  URL: https://issues.apache.org/jira/browse/JENA-897
>>>>>              Project: Apache Jena
>>>>>           Issue Type: Bug
>>>>>           Components: JDBC
>>>>>     Affects Versions: Jena 2.12.1, Jena 2.13.0
>>>>>          Environment: Windowx 8.0 x64, C: with 34 GB free
>>>>>             Reporter: Stian Soiland-Reyes
>>>>>             Priority: Critical
>>>>>              Fix For: Jena 2.13.1
>>>>>
>>>>>
>>>>> .. and thus mvn clean install on Windows will easily consume 37 GB on
>>>>> C: and run out of disk space - even if Jena is built on a larger
>>>>> partition.
>>>>
>>>>
>>>>
>>>> --
>>>> This message was sent by Atlassian JIRA
>>>> (v6.3.4#6332)
>>>
>>>
>>>
>>> --
>>> Stian Soiland-Reyes
>>> Apache Taverna (incubating), Apache Commons RDF (incubating)
>>> http://orcid.org/0000-0001-9842-9718
>>
>>
>>
>>
>
>
>