CQ-OPS

1) AEM JVM init Arguments Required for test environments

-Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8080 -Dcom.sun.management.jmxremote.rmi.port=8080 -Djava.rmi.server.hostname=101.102.103.104 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

For prod environments, you need to use authentication, and secure the transport with SSL.  Instructions here.

2) Firewall Changes in the OS

For RHEL 7.5, you’d need to run the following (assuming port is 8080)

firewall-cmd \-\-zone=public \-\-add-port=8080/tcp \-\-permanent firewall-cmd \-\-reload

3) Security Group Changes in the Cloud Provider (assuming port is 8080)

Apache Bench is useful for quick and dirty tests of AEM.  It is especially useful for testing the capacity of your load-balancer/Dispatcher tier.

For Windows, download Apache HTTP Server (64-bit) from Apache Lounge.  Apache Bench will be available in the /bin folder.  You can use ab.exe, or abs.exe for HTTPS requests.

The following command will send 20,000 GET requests at a concurrent user level of 400:

abs -r -k -n 20000 -c 400 -H "Accept-Encoding: gzip,deflate" https://stage.test.com/content/en.html

Use the 95th percentile value (reported in milliseconds) for decision making.

The following will upload a JPEG image to DAM at /content/dam/adobetest:

There are cases where you would want to tune AEM settings for higher workflow throughput after up-sizing the hardware.

Tuning Knob #1 : Max Parallel attribute of Job Queue

Navigate to /system/console/slingevent , edit the [Job Queue Configuration] for [Granite Workflow Queue], [Granite Transient Workflow Queue], [Granite Workflow External Process Job Queue] or any other job queue of type ‘Topic Round Robin’ that is relevant to your workload.  Increase the value for ‘Max Parallel’.  The default value is usually 0.5 (half the number of vCPUs of your instance).

Tuning Knob #2 : Apache Sling Job Thread Pool

Now that Adobe has fixed many of the problems with On-Line RevisionGC() and the associated On-Line tar Compaction in AEM 6.3 onwards, I notice that administrators are beginning to go easy on repository maintenance.

Note that Adobe still recommends monthly off-line maintenance.  This monthly planned downtime is a very good idea for two reasons:

1) Off-Line tar Compaction is still better at re-claiming wasted storage than its On-Line counterpart

2) The stop/start cycle will expose other repository integrity issues that crop up only when AEM is re-started (at the most inopportune time!).

So here are steps you can follow (you may end up adding others).

Make sure AEM is shut down.

On Linux, make sure you are running as the same user as the one that owns the AEM files.  If you run as root, remember to change file ownership back afterwards.

Make sure the oak-run.jar you use matches the version of Jackrabbit Oak AEM is on.

See this and this for more information.

Some of these steps can take many hours or even days.

1) Stop AEM

2) Re-index some (or all) key indexes with

java -Doak.index.ramBufferSizeMB=4096 -jar oak-run-1.8.9.jar index \-\-reindex \-\-index-paths=/oak:index/lucene,/oak:index/cqPageLucene,/oak:index/damAssetLucene,/oak:index/versionStoreIndex,/oak:index/workflowDataLucene \-\-read-write \-\-fds-path=G:\\AEM_6.4\\author\\crx-quickstart\\repository\\datastore  G:\\AEM_6.4\\author\\crx-quickstart\\repository\\segmentstore

3) Start AEM for the new indexes to be downloaded an cached locally

4) Stop AEM

5) Check the consistency of the Oak repository node store

java -jar oak-run-1.8.9.jar check \-\-bin G:\\AEM_6.4\\author\\crx-quickstart\\repository\\segmentstore

6) Remove any un-referenced checkpoints

java -jar oak-run-1.8.9.jar checkpoints G:\\AEM_6.4\\author\\crx-quickstart\\repository\\segmentstore rm-unreferenced

7) tar Compaction

java -Dtar.memoryMapped=true -Xmx8g -jar oak-run-1.8.9.jar compact G:\\AEM_6.4\\author\\crx-quickstart\\repository\\segmentstore

8) Move any .tar.bak and .tar.ro.bak files out of the /segmentstore folder to an archive folder somewhere (can be deleted later)

9)  Check the consistency of the Oak repository blob store, make sure no missing blobs are reported

java -jar oak-run-1.8.9.jar datastorecheck \-\-consistency \-\-repoHome G:\\AEM_6.4\\author\\crx-quickstart\\repository \-\-store G:\\AEM_6.4\\author\\crx-quickstart\\repository\\segmentstore \-\-fds G:\AEM_6.4\author\crx-quickstart\install\\org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config

10) If you ran the above commands as root, change file ownership back with

chown aem:aem -R /mnt/aem/author

Adobe official documentation for setting up AEM on Windows Server 2016 to auto-start on reboots is available here.  It uses Apache Commons Daemon Service Runner prunsrv_amd64.exe for this.  AEM ships with it (available at \crx-quickstart\opt\helpers).

Even though the documentation tells you that any changes you make in instsrv.bat will not take effect until you re-install the Windows service, this is not true.  You can also edit the registry key (as Administrator) Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\cq5\Parameters\Java and re-start the service for the changes to take effect.  For example, changing the heap size, or enabling GC logging.

Note that you can change the TCP port (-p argument) as well as the local network interfaces (-a argument) AEM LISTENs on by editing the registry key (as Administrator) HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\cq5\Parameters\Start\Params

A web page is not only an HTML page, but also lots of additional embedded resources such as images, web fonts, .css and .js files  A simple load test using curl will only request the HTML page, but not the embedded resources.  If you hard-code the additional calls to the embedded resources as curl commands, your code runs the risk of becoming obsolete quickly when the web page changes.  This latter issue also afflicts load testing tools that allow you to record all of the calls involved in the loading of single web page as a single, complete transaction (JMeter, LoadRunner etc.).

An obvious solution is to hire multiple people to use real browsers to test.  This is not practical.  Google Chrome got a headless feature in version 59 which can be invoked from the command line on instances without a windowing subsystem.  Watch this YouTube video.

On Windows 10, here’s the command that will load not only a we-retail web page, but also all of its embedded resources.  Note that the \-\-disable-gpu argument is only needed for Windows, not MacOS or Linux.  Verify by tailing AEM’s request.log

"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" \-\-headless \-\-disable-gpu http://localhost:4503/content/we-retail/us/en.html

You can now build yourself a decent and realistic load test by writing a bash shell script or a Windows PowerShell script to repeatedly load a set of web pages from a CSV file, and run this from multiple instances.  Here is how to invoke [Google Chrome Headless] in PowerShell:

Start-Process 'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe' "\-\-headless \-\-disable-gpu http://localhost:4503/content/we-retail/us/en.html"

A single instance can be used to generate multi-user load by simultaneously running multiple instances of PowerShell or bash.  Below is a table of the results of a 4-user, 1-hour load test on the complete set of we-retail pages (399 pages).  You can download the Excel spreadsheet here (note that it has multiple worksheets).

1) From the Adobe Experience Cloud GitHub repo, download the following:

a) aem-core-wcm-components b) aem-sample-we-retail

2) Build both with Maven (see this for more)

3) Deploy the following (order matters) to your AEM PUBLISH instance;

a) aem-core-wcm-components-master\all\target\core.wcm.components.all-2.0.7-SNAPSHOT.zip b) aem-core-wcm-components-master\extension\target\core.wcm.components.extension-1.0.3-SNAPSHOT.zip c) aem-sample-we-retail-master\all\target\we.retail.all-3.0.1-SNAPSHOT.zip

4) On the PUBLISH, navigate to http://localhost:4503/content/we-retail/us/en.html

Excellent webinar on its use by AEM Customer Care consultants Leo Berliant and Kunwarpeet Saluja available here.

Using CRX DE Lite, navigate to /libs/dam/gui/content/assets/fileupload and change the value of the JCR property ‘sizeLimit’ (2 GB is the default value in AEM 6.3) to the desired value (in bytes).

MongoDB comes with a free disk performance assessment tool named mongoperf that can be run independently without MongoDB.

1) Copy the contents of MongoDB’s /bin folder onto to the disk whose performance you want to assess

2) Then run it with the following command (the config file mongoperf.conf.js is available here)

mongoperf < D:\temp\mongoperf.conf.js

The “Developer Tools” that come packaged with Google Chrome, Firefox, Apple Safari and Microsoft Edge provide detailed sub-asset level information on web page load performance.  However, to get detailed information on each of those sub-assets in the page you would need some Apache Sling tooling.

Adobe’s David Gonzalez has built an “AEM Chrome plugin” that does this (obviously this only works in Google Chrome!).  You have to:

- install the plugin in the Google Chrome browser - provide AEM login credentials to the plugin - install an OSGi bundle (Sling Log Tracer) in AEM - start the OSGi bundle - configure the OSGi bundle and then enable it

He has made two YouTube videos about it:

1) Setup and Configuration 2) How to Use

If you have admin access to the AEM server, you can also enable the Apache Sling Request Progress Tracker Log Filter (/system/console/configMgr).

Click on image for a larger version.

Then, if you create a new Apache Sling Logging Logger configuration for org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter (DEBUG), it will spit out similar information into its own log file.

More here from Prakash Venkatesh of Perficient.

If you are new to Apache Sling, watch these videos:

Apache Sling source code resides in SVN repositories.  To download a Sling module such as the [Request Processing Analyzer] (reqanalyzer), you can perform the following steps:

1) Download and install TortoiseSVN (an Apache SVN client, implemented as a Windows shell extension)

2) Create a folder for the module (D:\Projects\Apache\Sling\reqanalyzer)

3) Obtain the module’s SVN URL (http://svn.apache.org/repos/asf/sling/trunk/contrib/extensions/reqanalyzer/)

3) Right-click mouse and choose SVN Checkout...