You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Andy Seaborne <an...@apache.org> on 2019/06/01 12:59:02 UTC
Re: Issue with TDB file locking
Hi Amit,
You are correct - when used in a JVM, the dataset remains manged by the
JVM after loading. This is intentional - the app may want to use the
dataset some more in Java code.
You can:
1/ Fork the process as a separate OS process using java.util.ProcessBuilder
2/ Force it out of the running JVM with
TDBInternal.expel(dataset)
or
StoreConenction.release(location)
3/ Work with the dataset in Fuseki - you can POST data to it
4/ Not use a war file but use Fuseki main, and run your program and
Fuseki in the same JVM.
Symbolic links make no difference - they are only involved in file
system path naming. Once opened, the OS pathname isn't involved.
Andy
On 31/05/2019 21:23, Amit Kumar wrote:
> Hi,
> I'm having issue an issue with using Fuseki as a war file inside jetty. We
> have a java application which looks for new data and loads it onto fuseki.
> When it is doing it for the first time, it has to create a new index, so we
> call the tdb2.tdbloader class directly with appropriate parameters. Here is
> the relevant code snippet
>
> import tdb2.tdbloader;
>
> ...
>
> public static List<String> TDBLoaderParams = Arrays.asList("-q",
>> "--loader", "parallel", "--loc");
>> ...
>> List<String> tdbParams = new ArrayList<>(TDBLoaderParams);
>> tdbParams.add(dstPath);
>> tdbParams.addAll(srcPaths);
>> try {
>> ...
>> tdbloader.main(tdbParams.toArray(new String[tdbParams.size()]));
>> }
>> ...
>> ...
>>
>
>
> This works as expected and create a new index at the desired location. Now,
> the problem is the same java process has to apply other feeds onto the same
> graph, so we move the softlink pointing to the fuseki database and execute
> a jetty restart command by calling a shell action using Java ProcessBuilder
> process.
> The jetty though fails to restart with the following error:
>
>>
>> org.apache.jena.assembler.exceptions.AssemblerException: caught: Failed to
>> get a lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock':
>> held by process 35918 at
>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:165)
>> at
>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:144)
>> at
>> org.apache.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:93)
>> at
>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:39)
>> at
>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:35)
>> at
>> org.apache.jena.fuseki.build.FusekiConfig.getDataset(FusekiConfig.java:353)
>> at
>> org.apache.jena.fuseki.build.FusekiConfig.buildDataService(FusekiConfig.java:303)
>> at
>> org.apache.jena.fuseki.build.FusekiConfig.buildDataAccessPoint(FusekiConfig.java:292)
>> at
>> org.apache.jena.fuseki.build.FusekiConfig.readConfiguration(FusekiConfig.java:275)
>> at
>> org.apache.jena.fuseki.build.FusekiConfig.readConfigurationDirectory(FusekiConfig.java:254)
>> ... ... Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get a
>> lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock': held by
>> process 35918 at
>> org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
>> at
>> org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
>> at
>> org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:88)
>> at
>> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
>> at
>> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
>>
>
> So the Jetty Server fails to restart while the jvm that created the index
> is still active, even though it has completed the index building part. It
> restarts fine once the loader JVM dies. It looks like the parallel loader
> class is not releasing the lock() properly. FYI, I'm using 3.11 Jena
>
> From apache jena github
> https://github.com/apache/jena/blob/66d14eda44dd0f37a9df31ed27c55c26600ff5f8/jena-db/jena-dboe-base/src/main/java/org/apache/jena/dboe/base/file/ProcessFileLock.java#L184-L218
>
>
>
> Regards
> Amit Kumar
>
Re: Issue with TDB file locking
Posted by Andy Seaborne <an...@apache.org>.
Summarising from Amit's reply:
On 01/06/2019 13:59, Andy Seaborne wrote:
> Hi Amit,
>
> You are correct - when used in a JVM, the dataset remains manged by the
> JVM after loading. This is intentional - the app may want to use the
> dataset some more in Java code.
>
> You can:
> 1/ Fork the process as a separate OS process using java.util.ProcessBuilder
>
> 2/ Force it out of the running JVM with
> TDBInternal.expel(dataset)
Works.
> or
> StoreConenction.release(location)
This does not work.
>
> 3/ Work with the dataset in Fuseki - you can POST data to it
>
> 4/ Not use a war file but use Fuseki main, and run your program and
> Fuseki in the same JVM.
>
> Symbolic links make no difference - they are only involved in file
> system path naming. Once opened, the OS pathname isn't involved.
>
> Andy
>
> On 31/05/2019 21:23, Amit Kumar wrote:
>> Hi,
>> I'm having issue an issue with using Fuseki as a war file inside
>> jetty. We
>> have a java application which looks for new data and loads it onto
>> fuseki.
>> When it is doing it for the first time, it has to create a new index,
>> so we
>> call the tdb2.tdbloader class directly with appropriate parameters.
>> Here is
>> the relevant code snippet
>>
>> import tdb2.tdbloader;
>>
>> ...
>>
>> public static List<String> TDBLoaderParams = Arrays.asList("-q",
>>> "--loader", "parallel", "--loc");
>>> ...
>>> List<String> tdbParams = new ArrayList<>(TDBLoaderParams);
>>> tdbParams.add(dstPath);
>>> tdbParams.addAll(srcPaths);
>>> try {
>>> ...
>>> tdbloader.main(tdbParams.toArray(new String[tdbParams.size()]));
>>> }
>>> ...
>>> ...
>>>
>>
>>
>> This works as expected and create a new index at the desired location.
>> Now,
>> the problem is the same java process has to apply other feeds onto the
>> same
>> graph, so we move the softlink pointing to the fuseki database and
>> execute
>> a jetty restart command by calling a shell action using Java
>> ProcessBuilder
>> process.
>> The jetty though fails to restart with the following error:
>>
>>>
>>> org.apache.jena.assembler.exceptions.AssemblerException: caught:
>>> Failed to
>>> get a lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock':
>>> held by process 35918 at
>>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:165)
>>>
>>> at
>>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:144)
>>>
>>> at
>>> org.apache.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:93)
>>>
>>> at
>>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:39)
>>>
>>> at
>>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:35)
>>>
>>> at
>>> org.apache.jena.fuseki.build.FusekiConfig.getDataset(FusekiConfig.java:353)
>>>
>>> at
>>> org.apache.jena.fuseki.build.FusekiConfig.buildDataService(FusekiConfig.java:303)
>>>
>>> at
>>> org.apache.jena.fuseki.build.FusekiConfig.buildDataAccessPoint(FusekiConfig.java:292)
>>>
>>> at
>>> org.apache.jena.fuseki.build.FusekiConfig.readConfiguration(FusekiConfig.java:275)
>>>
>>> at
>>> org.apache.jena.fuseki.build.FusekiConfig.readConfigurationDirectory(FusekiConfig.java:254)
>>>
>>> ... ... Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to
>>> get a
>>> lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock': held by
>>> process 35918 at
>>> org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
>>>
>>> at
>>> org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
>>>
>>> at
>>> org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:88)
>>>
>>> at
>>> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
>>>
>>> at
>>> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
>>>
>>>
>>
>> So the Jetty Server fails to restart while the jvm that created the index
>> is still active, even though it has completed the index building part. It
>> restarts fine once the loader JVM dies. It looks like the parallel
>> loader
>> class is not releasing the lock() properly. FYI, I'm using 3.11 Jena
>>
>> From apache jena github
>> https://github.com/apache/jena/blob/66d14eda44dd0f37a9df31ed27c55c26600ff5f8/jena-db/jena-dboe-base/src/main/java/org/apache/jena/dboe/base/file/ProcessFileLock.java#L184-L218
>>
>>
>>
>>
>> Regards
>> Amit Kumar
>>
Re: Issue with TDB file locking
Posted by Andy Seaborne <an...@apache.org>.
On 05/06/2019 20:42, Amit Kumar wrote:
> Hi Andy,
> Thanks for your reply. I tried your 2nd Option.
> The StoreConnection.release(location) option didn't seem to work.
> I'm assuming the location should point to the main TopLevel Directory
>
> /etc/fuseki/databases/201906031724/
> ├── Data-0001
> │ ├── GOSP.bpt
> │ ├── GOSP.dat
> | | ...
> │ └── tdb.lock
> └── tdb.lock
>
> I gave "/etc/fuseki/databases/201906031724" as location input
Checking the code, I see that for StoreConnection, it'll be
/etc/fuseki/databases/201906031724/Data-0001 (StoreConnection in TDB2
manages the the current dataset; TDBInternal.expel understands
switchable datasets - the level up at /etc/fuseki/databases/201906031724/.
Too much TDB1 thinking!
> private boolean postProcessingWebServer(String path) {
>
> Location location = Location.create(path);
>
> String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
>
> LOGGER.info("location {} is presence in StoreConnection.cache
> is {}", lockFilename, StoreConnection.isSetup(location));
>
> StoreConnection.release(location);
>
> return ExecUtil.restartJetty();
>
> }
>
>
> INFO location /etc/fuseki/databases/201906031724/tdb.lock is presence in
> StoreConnection.cache is false
>
> This option failed with the same original error. The storeConnection has a
> cache, and this location was not in that cache().
>
> =============
> So next I tried the following option
>
> private boolean postProcessingWebServer(String path) {
> Location location = Location.create(path);
> String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
> LOGGER.info("location of the lock file is {}", lockFilename);
> StoreConnection storeConnection =
> StoreConnection.connectCreate(location);
> storeConnection.getLock().unlock();
> return ExecUtil.restartJetty();
> }
>
>
> This time I got the following error
>
> org.apache.jena.dboe.base.file.AlreadyLocked: Failed to get a lock:
> file='/etc/fuseki/databases/201906031727/tdb.lock': Lock already held
> at org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:187)
> at org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
> at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
> at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
> at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:52)
>
>
> ========
> Then I tried to use the ProcessFileLock directly
>
> private boolean postProcessingWebServer(String path) {
> Location location = Location.create(path);
> String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
> ProcessFileLock fileLock = ProcessFileLock.create(lockFilename);
> LOGGER.info("Lock exists for {} is {}", lockFilename,
> fileLock.isLockedHere());
> fileLock.unlock();
> return ExecUtil.restartJetty();
> }
>
> INFO Lock exists for /etc/fuseki/databases/201906031755/tdb.lock is true
>
> Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get a
> lock: file='/etc/fuseki/databases/201906031755/Data-0001/tdb.lock':
> held by process 8211
> at org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
> at org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
> at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
> at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
> at org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:97)
> at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:78)
> at org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:90)
> at org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
> at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
> at org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:76)
> at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:61)
> at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:52)
>
> This time the error was for the lock file which is inside the Data-0001
> folder instead.
> =====
> When I tried using the unlock on both the lock file explicitly, it worked.
> I also tried the TDBInternal.expel(dataset) and it worked
>
> private boolean postProcessingWebServer(String path) {
> Location location = Location.create(path);
> DatabaseConnection databaseConnection =
> DatabaseConnection.connectCreate(location);
> TDBInternal.expel(databaseConnection.getDatasetGraph());
> return ExecUtil.restartJetty();
> }
>
>
>
> I'm going ahead with the TDBInternal.expel option for now.
>
> Regards
> Amit
>
> On Sat, Jun 1, 2019 at 5:59 AM Andy Seaborne <an...@apache.org> wrote:
>
>> Hi Amit,
>>
>> You are correct - when used in a JVM, the dataset remains manged by the
>> JVM after loading. This is intentional - the app may want to use the
>> dataset some more in Java code.
>>
>> You can:
>> 1/ Fork the process as a separate OS process using java.util.ProcessBuilder
>>
>> 2/ Force it out of the running JVM with
>> TDBInternal.expel(dataset)
>> or
>> StoreConenction.release(location)
>>
>> 3/ Work with the dataset in Fuseki - you can POST data to it
>>
>> 4/ Not use a war file but use Fuseki main, and run your program and
>> Fuseki in the same JVM.
>>
>> Symbolic links make no difference - they are only involved in file
>> system path naming. Once opened, the OS pathname isn't involved.
>>
>> Andy
>>
>> On 31/05/2019 21:23, Amit Kumar wrote:
>>> Hi,
>>> I'm having issue an issue with using Fuseki as a war file inside jetty.
>> We
>>> have a java application which looks for new data and loads it onto
>> fuseki.
>>> When it is doing it for the first time, it has to create a new index, so
>> we
>>> call the tdb2.tdbloader class directly with appropriate parameters. Here
>> is
>>> the relevant code snippet
>>>
>>> import tdb2.tdbloader;
>>>
>>> ...
>>>
>>> public static List<String> TDBLoaderParams = Arrays.asList("-q",
>>>> "--loader", "parallel", "--loc");
>>>> ...
>>>> List<String> tdbParams = new ArrayList<>(TDBLoaderParams);
>>>> tdbParams.add(dstPath);
>>>> tdbParams.addAll(srcPaths);
>>>> try {
>>>> ...
>>>> tdbloader.main(tdbParams.toArray(new String[tdbParams.size()]));
>>>> }
>>>> ...
>>>> ...
>>>>
>>>
>>>
>>> This works as expected and create a new index at the desired location.
>> Now,
>>> the problem is the same java process has to apply other feeds onto the
>> same
>>> graph, so we move the softlink pointing to the fuseki database and
>> execute
>>> a jetty restart command by calling a shell action using Java
>> ProcessBuilder
>>> process.
>>> The jetty though fails to restart with the following error:
>>>
>>>>
>>>> org.apache.jena.assembler.exceptions.AssemblerException: caught: Failed
>> to
>>>> get a lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock':
>>>> held by process 35918 at
>>>>
>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:165)
>>>> at
>>>>
>> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:144)
>>>> at
>>>>
>> org.apache.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:93)
>>>> at
>>>>
>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:39)
>>>> at
>>>>
>> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:35)
>>>> at
>>>>
>> org.apache.jena.fuseki.build.FusekiConfig.getDataset(FusekiConfig.java:353)
>>>> at
>>>>
>> org.apache.jena.fuseki.build.FusekiConfig.buildDataService(FusekiConfig.java:303)
>>>> at
>>>>
>> org.apache.jena.fuseki.build.FusekiConfig.buildDataAccessPoint(FusekiConfig.java:292)
>>>> at
>>>>
>> org.apache.jena.fuseki.build.FusekiConfig.readConfiguration(FusekiConfig.java:275)
>>>> at
>>>>
>> org.apache.jena.fuseki.build.FusekiConfig.readConfigurationDirectory(FusekiConfig.java:254)
>>>> ... ... Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get
>> a
>>>> lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock': held by
>>>> process 35918 at
>>>>
>> org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
>>>> at
>>>>
>> org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
>>>> at
>>>>
>> org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:88)
>>>> at
>>>>
>> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
>>>> at
>>>>
>> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
>>>>
>>>
>>> So the Jetty Server fails to restart while the jvm that created the index
>>> is still active, even though it has completed the index building part. It
>>> restarts fine once the loader JVM dies. It looks like the parallel
>> loader
>>> class is not releasing the lock() properly. FYI, I'm using 3.11 Jena
>>>
>>> From apache jena github
>>>
>> https://github.com/apache/jena/blob/66d14eda44dd0f37a9df31ed27c55c26600ff5f8/jena-db/jena-dboe-base/src/main/java/org/apache/jena/dboe/base/file/ProcessFileLock.java#L184-L218
>>>
>>>
>>>
>>> Regards
>>> Amit Kumar
>>>
>>
>
Re: Issue with TDB file locking
Posted by Amit Kumar <as...@gmail.com>.
Hi Andy,
Thanks for your reply. I tried your 2nd Option.
The StoreConnection.release(location) option didn't seem to work.
I'm assuming the location should point to the main TopLevel Directory
/etc/fuseki/databases/201906031724/
├── Data-0001
│ ├── GOSP.bpt
│ ├── GOSP.dat
| | ...
│ └── tdb.lock
└── tdb.lock
I gave "/etc/fuseki/databases/201906031724" as location input
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
LOGGER.info("location {} is presence in StoreConnection.cache
is {}", lockFilename, StoreConnection.isSetup(location));
StoreConnection.release(location);
return ExecUtil.restartJetty();
}
INFO location /etc/fuseki/databases/201906031724/tdb.lock is presence in
StoreConnection.cache is false
This option failed with the same original error. The storeConnection has a
cache, and this location was not in that cache().
=============
So next I tried the following option
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
LOGGER.info("location of the lock file is {}", lockFilename);
StoreConnection storeConnection =
StoreConnection.connectCreate(location);
storeConnection.getLock().unlock();
return ExecUtil.restartJetty();
}
This time I got the following error
org.apache.jena.dboe.base.file.AlreadyLocked: Failed to get a lock:
file='/etc/fuseki/databases/201906031727/tdb.lock': Lock already held
at org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:187)
at org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:52)
========
Then I tried to use the ProcessFileLock directly
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
ProcessFileLock fileLock = ProcessFileLock.create(lockFilename);
LOGGER.info("Lock exists for {} is {}", lockFilename,
fileLock.isLockedHere());
fileLock.unlock();
return ExecUtil.restartJetty();
}
INFO Lock exists for /etc/fuseki/databases/201906031755/tdb.lock is true
Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get a
lock: file='/etc/fuseki/databases/201906031755/Data-0001/tdb.lock':
held by process 8211
at org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
at org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
at org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:97)
at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:78)
at org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:90)
at org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:76)
at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:61)
at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:52)
This time the error was for the lock file which is inside the Data-0001
folder instead.
=====
When I tried using the unlock on both the lock file explicitly, it worked.
I also tried the TDBInternal.expel(dataset) and it worked
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
DatabaseConnection databaseConnection =
DatabaseConnection.connectCreate(location);
TDBInternal.expel(databaseConnection.getDatasetGraph());
return ExecUtil.restartJetty();
}
I'm going ahead with the TDBInternal.expel option for now.
Regards
Amit
On Sat, Jun 1, 2019 at 5:59 AM Andy Seaborne <an...@apache.org> wrote:
> Hi Amit,
>
> You are correct - when used in a JVM, the dataset remains manged by the
> JVM after loading. This is intentional - the app may want to use the
> dataset some more in Java code.
>
> You can:
> 1/ Fork the process as a separate OS process using java.util.ProcessBuilder
>
> 2/ Force it out of the running JVM with
> TDBInternal.expel(dataset)
> or
> StoreConenction.release(location)
>
> 3/ Work with the dataset in Fuseki - you can POST data to it
>
> 4/ Not use a war file but use Fuseki main, and run your program and
> Fuseki in the same JVM.
>
> Symbolic links make no difference - they are only involved in file
> system path naming. Once opened, the OS pathname isn't involved.
>
> Andy
>
> On 31/05/2019 21:23, Amit Kumar wrote:
> > Hi,
> > I'm having issue an issue with using Fuseki as a war file inside jetty.
> We
> > have a java application which looks for new data and loads it onto
> fuseki.
> > When it is doing it for the first time, it has to create a new index, so
> we
> > call the tdb2.tdbloader class directly with appropriate parameters. Here
> is
> > the relevant code snippet
> >
> > import tdb2.tdbloader;
> >
> > ...
> >
> > public static List<String> TDBLoaderParams = Arrays.asList("-q",
> >> "--loader", "parallel", "--loc");
> >> ...
> >> List<String> tdbParams = new ArrayList<>(TDBLoaderParams);
> >> tdbParams.add(dstPath);
> >> tdbParams.addAll(srcPaths);
> >> try {
> >> ...
> >> tdbloader.main(tdbParams.toArray(new String[tdbParams.size()]));
> >> }
> >> ...
> >> ...
> >>
> >
> >
> > This works as expected and create a new index at the desired location.
> Now,
> > the problem is the same java process has to apply other feeds onto the
> same
> > graph, so we move the softlink pointing to the fuseki database and
> execute
> > a jetty restart command by calling a shell action using Java
> ProcessBuilder
> > process.
> > The jetty though fails to restart with the following error:
> >
> >>
> >> org.apache.jena.assembler.exceptions.AssemblerException: caught: Failed
> to
> >> get a lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock':
> >> held by process 35918 at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:165)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:144)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:93)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:39)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:35)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.getDataset(FusekiConfig.java:353)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.buildDataService(FusekiConfig.java:303)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.buildDataAccessPoint(FusekiConfig.java:292)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.readConfiguration(FusekiConfig.java:275)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.readConfigurationDirectory(FusekiConfig.java:254)
> >> ... ... Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get
> a
> >> lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock': held by
> >> process 35918 at
> >>
> org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
> >> at
> >>
> org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
> >> at
> >>
> org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:88)
> >> at
> >>
> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
> >> at
> >>
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
> >>
> >
> > So the Jetty Server fails to restart while the jvm that created the index
> > is still active, even though it has completed the index building part. It
> > restarts fine once the loader JVM dies. It looks like the parallel
> loader
> > class is not releasing the lock() properly. FYI, I'm using 3.11 Jena
> >
> > From apache jena github
> >
> https://github.com/apache/jena/blob/66d14eda44dd0f37a9df31ed27c55c26600ff5f8/jena-db/jena-dboe-base/src/main/java/org/apache/jena/dboe/base/file/ProcessFileLock.java#L184-L218
> >
> >
> >
> > Regards
> > Amit Kumar
> >
>