Application tuning is an activity which which every developer and architect will encounter sooner or later for any application. AEM based applications are no different. One of the activities that is particularly important with AEM 6
is query tuning. AEM 6 uses Jackrabbit OAK
as the repository and hence we would have to know more about OAK to tune the queries. Apart from tuning the queries itself a majority of the time you would be tuning your indexes based on the query. AEM itself uses a lot of queries within for the application to function. It is important to understand how the queries and the indexes work in AEM.
Update: This post was not written as a generic performance tuning guideline for AEM, but just on tuning the queries on OAK repository. Although a lot of the performance depends on how well tuned is your repository. Adobe's recent service packs on AEM 6, and AEM 6.1 in general are much stable and performant. Please refer to the Adobe's documentation for general guidelines on performance tuning.
While the instructions herein are primarily written for AEM applications, non-AEM applications which uses Jackrabbit OAK can also benefit.
There are no hard and fast rules that a particular index will work. This is very similar to database indexes. You would have to look at the current queries being fired and the current repository state and arrive at the best indexes. Often you arrive at the best indexes over time. You would have to apply an index and the re-evaluate your queries, then further tune your queries or try out a different index. Repeat these steps over and over again to find the best and optimized index.
However, there are some basic guidelines which should be followed, which can help you define the best indexes to you application specific needs:
Ground Rules (AEM Specific)
- Apply the latest Service Pack available and Oak specific hotfixes (if applicable). As of date SP-2 is publicly available.
- AEM has some built in indexes as well. Do not arbitrarily re-index these OOTB indexes. Indexing is a costly and resource-intensive operation. OOTB indexes should be re-indexed only when specifically advised by Adobe (this is a catch - sometimes you know that re-indexing a particular OOTB index will solve your problem at hand, but get a buy-in from Adobe for this). At other times you would have to create a new index (type) since the OOTB / existing index is in-efficient.
- Be aware that there has been some important changes since OAK 1.0.9 - especially related to Lucene indexes, hence be mindful about reference resources that advice about using OAK indexes. Check the OAK's latest documenting on query and indexing.
You can also use the OSGi Service of org.apache.jackrabbit.api.jmx.QueryStatManagerMBean to programmatically find the slow and popular queries.
- If you are using AEM, the best tool to help you debug and explain your queries, find out slow and popular queries is to install Adobe AEM ACS Tools package (0.0.20 and above). Once installed, you can have access it using the URL: http://yourinstance/etc/acs-tools/explain-query.html
- If the above is not possible, you may have to depend on JMX to find slow and popular queries, you can use the Felix console to identify slow and popular queries on any instance
Use the following debug logs for details on various aspects of query and indexing:
Use JMX to have high level overview about lucene indexes (OOTB or custom):
To look at the actual indexes, you can stop the AEM / Oak instance and use the oak-run "console" to explore the indexes among various other options:
- Enable DEBUG logging on org.apache.jackrabbit.oak.query for logs to find the details on query cost calculation (explain)
- Enable DEBUG logging on org.apache.jackrabbit.oak.plugins.index for logs to find the details on indexing
java -XX:MaxPermSize=512m -classpath luke-with-deps.jar;oak-lucene-1.1.6.jar org.getopt.luke.Luke
- The following jars needs to be used:
The basic steps for tuning your queries can be summarized as follows:
- Identify the slow queries in your system. This could be the one that takes a lot of time to execute primarily because there are not proper indexes and there are node traversals happening. The system itself can be queries through JMX to identify the slow and popular queries.
- Tune your queries to ensure that you can use indexes whenever possible.
- Identify properties and nodes that needs to be indexed, based on the slow queries identified, and create Oak indexes for them.
- Verify your queries and indexes after creating indexes. The indexes are optimum if the cost of the queries identified are lowest - 0 being the best.
- Repeat steps 3 and 4 above (by even creating different indexes for the same query) till the most cost effective index is created.
Identification of Slow Queries
Identify the slow queries by monitoring the error.log. Note the queries which traverses many nodes, where the log may look something like:
org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed 1000 nodes
... consider creating an index or changing the query
You can also identify the slowest and popular queries in the system (which the Jackrabbit Oak keeps track of) using JMX or Adobe AEM Commons as described in the "Tools" section above.
For the same query you can create multiple indexes. However choose the one to be promoted to your production instances which prove to be the the best in terms of achieving lower cost to the query at hand.
Consider the following when deciding the type of index to be used:
- Full Text Index Types
Lucene Property Index
- Lucene & Solr
- Aggregate Index - query time aggregation
- Since OAK-1.0.9 Lucene Property indexes are preferred indexes over other type of indexes
- Combines properties & full-text.
- Supports property condition, range conditions, ordering, full-text
- Index-time aggregation - performance improvements
- Using Lucene & Solr.
- Follow instructions for creating the index nodes as in the OAK's Lucene documentation.
- Default Indexes
- Unique or Non-Unique
- eg. UUID
- property equals query works best
- Internally Uses Property Index
- Uses primaryType and mixins to identify nodes
- Traverses Repo
- Does not store any data
- Used to retrieve the given path or its child nodes (sometimes the parent nodes as well)
- Not recommended of nodes > 1000 nodes
- Currently, the indexes can be found under /oak:index/ on AEM.
- Once you have identified the best indexes, plan to install them as a package on AEM.
- Re-indexing is usually not needed. However when you see degrade in system performance, do double check your existing indexes if it would need a re-index.
- Re-index is a costly and resource intensive operation. Use it only when any oak upgrade or mentioned explicitly in release doc for any patch or hotfix
- Avoid having multiple properties in index , it may slow your query , its recommended to have one index for each property.
- Set the system property oak.queryLimitInMemory to 200000 (or something similar). This will limit the traversal of the nodes to that number
Jackrabbit Oak is a project under active development, especially around the area of Indexing. Hence please check official documentation for details. Indexing and tuning requires a lot of observation and practice to get perfect.
I would recommend watching a webinar on this topic
by Thomas Muller and Marcel Reutegger on AEM Gems, which is a bit old (before OAK-1.0.9 was released), but immensely useful to understand indexing on Jackrabbit Oak.