This section is intended for system administrators that are already familiar with BigOWLIM and the Sesame openRDF infrastructure, who wish to configure their system for optimal performance. Best performance is typically measured by the shortest load time and fastest query answering. Many factors affect the performance of a BigOWLIM repository in many different ways. This section is an attempt to bring together all factors that affect performance, however it is measured.
Memory configuration is the single most important factor for optimising the performance of BigOWLIM. In every respect, the more memory available the better the performance. The only question, is how to divide up the available memory between the various BigOWLIM data structures in order to achieve the best overall behaviour.
The maximum amount of heap space used by a Java virtual machine (JVM) is specified using the -Xmx virtual machine parameter. The value should be no higher than the amount of free memory available in the target system multiplied by some factor to allow for extra runtime overhead, say approximately ~90%.
The heap space available is used by:
Simplistically, the memory required for storing entities is determined by the number of entities in the dataset, where the memory required is 4 bytes per slot allocated by entity-index-size plus 12 bytes for the actual number of entities.
A more complete view of memory use for BigOWLIM is given here:
During the initial load, some speed up can be achieved by switching journaling off, by setting the journaling configuration parameter to false. All this does is to stop the transaction log being written to disk (in case of failure), but a noticeable speed up is possible. During normal use, it is recommended to enable journaling for more reliable recovery after an unexpected termination.
If the BigOWLIM repository is being hosted by the Sesame HTTP servlet then the maximum heap space will apply to the servlet container (tomcat). In which case, allow some more heap memory for the runtime overhead, especially if running at the same time as other servlets. Also, some configuration of the servlet container might improve performance, e.g. increasing the permanent generation, which by default is 64MB. Quadrupling (for tomcat) with -XX:MaxPermSize=256m might help. Further information can be found in the tomcat documentation.
BigOWLIM's inference policy is one of total materialisation, where implicit statements are inferred from explicit statements as soon as they are inserted in to the repository suing the specified semantics ruleset. This approach has the advantage that query answering can be achieved very quickly, since no inference needs to be done at query time.
The algorithm used to identify and remove those inferred statements that can no longer be derived using the explicit statements being deleted is as follows:
The difficulty with the current algorithm is that almost all delete operations will follow inference paths that touch schema statements, which then lead to almost all other statements in the repository. This can lead to smooth delete taking a very long time indeed.
What can stop the algorithm touching too many (possibly all) statements, however, is that the algorithm will not go further if a visited statement is marked read-only. Since a read-only statement cannot be deleted, there is no reason to find what statements are inferred from it (such inferred statements might still get deleted, but they will be found by following other inference paths).
Consider the following statements:
When using the owl-horst rule set the removal of the statement:
will cause the following sequence of events:
Statements [<Reviewer40476> rdf:type owl:Thing], etc, exist because of the statements [<Reviewer40476> rdf:type <MyClass>] and [<MyClass> rdfs:subClassOf owl:Thing].
In large datasets there are typically millions of statements [X rdf:type owl:Thing], and they will all be visited by the algorithm.
Predicate lists are two indices (SP and OP) that can improve performance in two separate situations:
As a rough guideline, a dataset with more than about 1000 predicates will benefit from using these indices for both loading and query answering. Predicate list indices are not enabled by default, but can be switched on using the enablePredicateList configuration parameter.
Two further indices can also be used for providing better performance when executing queries that use context and/or tripleset ids. (Tripleset ids are not exposed through the Sesame interface, so can be ignored).
BigOWLIM uses a number of query optimisation techniques by default. These can be disabled by using the enable-optimization configuration parameter set to false, however there is rarely any need to do this.
This optimization applies when the repository contains a large number of literals with language tags and it is necessary to execute queries that filter based on language, e.g. using the following SPARQL query construct:
The presence of many owl:sameAs statements – such as when using several LOD datasets and link sets – causes an explosion in the number of inferred statements. For a simple example, if A is a city in country X, and B and C are alternative names for A, and Y an Z are alternative names for X, then the inference engine should infer: B in X, C in X, B in Y, C in Y, B in Z, C in Z also.
During query answering, all members of each equivalence appearing in a query are substituted to generate sound and complete query results. However, even though the mechanism to store equivalences is standard and cannot be switched off, it is possible to prevent the enumeration of equivalence classes during query answering. When using a dataset with many owl:sameAs statements, turning of the enumeration can dramatically reduce the number of duplicated query results, where a single URI from each equivalence class is chosen to be representative.
Wheras the same query with the pseudo-graph that prevents equivalence class enumeration:
The complexity of the rule set has a large effect on loading performance and on the overall size of the repository after loading. The complexity of the standard rule sets increases as follows:
It should be noted that all rules affect the loading speed, even if they never actually infer any new statements. This is because as new statements are added, they are pushed in to every rule to see if anything is inferred. Often this can result in many joins being computed even though the rule never 'fires'.
If better load performance is required and it is known that the dataset does not contain anything that will apply to certain rules then they can be omitted from the ruleset. To do this, copy the appropriate '.pie' file included in the distribution and remove the unused rules. Then set the ruleset configuration parameter to the full pathname to this custom rule set.
SwiftOWLIM version 2.9 contained a special optimisation that prevents the materialisation of inferred statements as the result of transitive chains. Instead, these inferences were computed during query answering.
The life-cycle of a repository instance typically starts with the initial loading of datasets followed by the processing of queries and updates. The loading of a large dataset can take a long time - 12 hours for a billion statements with inference is not unusual. Therefore, it is often useful to use a different configuration during loading than during normal use. Furthermore, if a dataset is frequently loaded, because it changes gradually over time, then the loading configuration can be evolved as the administrator gets more familiar with the behaviour of BigOWLIM with this dataset. Many properties of the dataset only become apparent after the initial load (such as the number of unique entities) and this information can be used to optimise the loading step the next time round or to improve the normal use configuration.
A typical initialisation life-cycle would be like this:
Unless the repository needs to answer queries during the initialisation phase, the repository can be configured with the minimum number of options and indices, with a large portion of the available heap space given over to the cache memory:
The optional indices can be built at a later time when the repository is used for query answering. The details of all optional indices, caches and optimisations have been covered previously in this document. Some experimentation is required using typical query patterns from the user environment.
If any one of these is missed out, it will be calculated. If two or more are unspecified then the remaining cache memory is divided evenly between them.
Skip to end of metadata Go to start of metadata