[GitHub] [jena] rvesse commented on a change in pull request #1164: JENA-2208: Improve TDB2 xloader

[GitBox <gi...@apache.org>]

rvesse commented on a change in pull request #1164:
URL: https://github.com/apache/jena/pull/1164#discussion_r785817594



##########
File path: jena-db/jena-tdb2/src/main/java/org/apache/jena/tdb2/xloader/BulkLoaderX.java
##########
@@ -41,6 +56,26 @@
      */
     public static boolean CompressSortIndexFiles = true;
 
+    // Ubuntu: it now (21.04) is at /usr/bin/gzip.
+    //   /bin has become a symbolic link to /usr/bin.
+    //   New installs of 20.04 have it at /usr/bin, upgrades have it at /bin.
+    /*package*/ static String gzipProgram() {
+        if ( programInstalledAt("/usr/bin/gzip") )

Review comment:
       Since `programInstalledAt` throws an exception if the given path is not executable won't this never try the other option in that case?

##########
File path: jena-db/jena-tdb2/src/main/java/org/apache/jena/tdb2/xloader/ProcIngestDataX.java
##########
@@ -103,9 +117,41 @@ public static void exec(String location,
         SystemIRIx.setProvider(provider);
     }
 
+    private static DatasetGraph getDatasetGraph(String location) {
+        Location loc = Location.create(location);
+        // Ensure reset
+        DatasetGraph dsg0 = DatabaseMgr.connectDatasetGraph(location);
+        TDBInternal.expel(dsg0);
+
+        StoreParams dftStoreParams = StoreParams.getDftStoreParams();
+        StoreParams storeParams = dftStoreParams;
+
+        if ( true ) {
+            storeParams = StoreParams.builder(StoreParams.getDftStoreParams())
+                    .node2NodeIdCacheSize(10_000_000)
+                    .build();
+        }
+
+        DatasetGraph dsg = DatabaseConnection.connectCreate(loc, storeParams).getDatasetGraph();
+        StoreParams storeParamsActual = TDBInternal.getDatasetGraphTDB(dsg).getStoreParams();
+
+        FmtLog.info(LOG, "Node to NodeId cache size: %,d", storeParamsActual.getNode2NodeIdCacheSize());
+        FmtLog.info(LOG, "NodeId to Node cache size: %,d", storeParamsActual.getNodeId2NodeCacheSize());
+
+        return dsg;
+    }
+
     private static Pair<Long, Long> build(DatasetGraph dsg, ProgressMonitor monitor,
                               OutputStream outputTriples, OutputStream outputQuads,
                               List<String> datafiles) {
+
+        int node2NodeIdCacheSize = StoreParamsConst.Node2NodeIdCacheSize; // 200k
+        node2NodeIdCacheSize = 1_000_000_000;// 1B!
+
+        StoreParams storeParmas =
+                StoreParams.builder().node2NodeIdCacheSize(node2NodeIdCacheSize)
+                .build();

Review comment:
       This set of `storeParams` doesn't obviously get used AFAICT




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@jena.apache.org
For additional commands, e-mail: pr-help@jena.apache.org