Dynamic Cache in App Service
A while ago, while I was browsing Kudu's setting documentation I stumbled upon a setting that caught my eye - Dynamic Cache. I have never heard or read about it in Microsoft's own documention (except that page on GitHub) so I decided to try it out.
The initial problem with App Service is its slow filesystem (which I have blogged about already more than once). Up until this discovery you had two options - either have all the files delivered through the slow replicated filesystem or enable Local Cache which results in the files written locally no longer being persistent which is a major issue with most PHP applications (even for full WordPress functionality - installing plugins, updates, etc.). Now, there is a third option, which is called Dynamic Cache. As per the documentation, it is available in two modes (copied from docs):
Full content caching: caches both file content and directory/file metadata (timestamps, size, directory content):
WEBSITE_DYNAMIC_CACHE=1
Directory metadata caching: will not cache content of the files, only the directory/file metadata (timestamps, size, directory content). That results in much less local disk use:
WEBSITE_DYNAMIC_CACHE=2
In other words, it appears that Dynamic Cache is a cache at the read layer, which boosts the read operations and write operations are unaffected, this going still through the network. Now knowing this, time to test the performance of these settings!
For the purpose of this test, I will be using a simple PHP script to read from the filesystem (1oo times 1000 file_get_contents operations). The site along with the storage has been hit for multiple times before the actual test (so that the cache - if used - could build and initialize properly).
<?php
echo "<h1>Performance Result</h1>";
$websiteDynamicCache = $_SERVER["WEBSITE_DYNAMIC_CACHE"];
echo "<b>WEBSITE_DYNAMIC_CACHE=$websiteDynamicCache</b>";
$averages = [];
for($j = 0; $j < 10; $j++) {
$startTime = microtime(true);
for($i = 0; $i < 1000; $i++){
// file_put_contents("sampledata/$i.txt", "test$i");
file_get_contents("sampledata/$i.txt");
}
$timeEnd = microtime(true);
$timeResult = $timeEnd - $startTime;
$averages[] = $timeResult;
}
$average = array_sum($averages) / count($averages);
$min = min($averages);
$max = max($averages);
echo "<br>Average: $average";
echo "<br>Minimum: $min";
echo "<br>Maximum: $max";
Starting with the default App Service settings - Dynamic Cache is disabled.
WEBSITE_DYNAMIC_CACHE=0
Average: 6.2657294273376
Minimum: 4.5602788925171
Maximum: 10.892529010773
This is actually quite high score, it took 6.3 seconds on average to read 1000 files from the filesystem! This is the original and default behavior of App Service, so you are likely to get these times everywhere. After running this multiple times, the times really vary (I chose the best run) all the way up to 26 seconds. This is obviously caused by the network latency - every request has to go to the storage worker and Azure Blob Storage on the background which can be time expensive.
Next, we are going with Dynamic Cache enabled set to mode 2 - caching everything except file contents, which means, that there are still reads happening over the network, however, file metadata, stamps etc. are all cached locally.
WEBSITE_DYNAMIC_CACHE=2
Average: 2.2721150946617
Minimum: 2.114581823349
Maximum: 3.6030380725861
You can obviously see the difference here. We have got 2.1 times better performance on average than with using the network storage, this is great improvement!
Last experiment to try is with Dynamic Cache enabled in contents caching mode.
WEBSITE_DYNAMIC_CACHE=1
Average: 0.61150848388672
Minimum: 0.5042462348938
Maximum: 0.85975289344788
Wow! We have got over 10 times better performance than with networked calls! The obvious question that comes to my mind is - why isn't the Dynamic Cache on by default?
It appears that since the early mentions (from 2017), Dynamic Cache can cause some trouble with specific applications running on App Service. Those applications also include PHP apps. However, having this enabled on a production WordPress site for over a month, we didn't notice any errors, but the performance has much improved since!
So are there any trade offs? Obviously, yes. Since files are cached for reads on the instances, they don't get refreshed the second you change them across all instances in the App Service Plan. From my tests, it usually took around 10 seconds for the change to be propagated to the end node - if you are using a single node App Service Plan, you should be fine, except the changes from Kudu site to your web site take 10 seconds as well. It appears that the Kudu site is hooked to the storage directly just like with Local Cache.
It is quite interesting that this feature is not mentioned in Microsoft's documentation, since it appears to have a really positive impact on the site performance. I found mentions of this feature as early as in 2017, which means it has been around for a while.
So if you are experiencing performance issues on App Service on Windows, I encourage you to go ahead and try the Dynamic Cache and see if it helps you app. However, keep on mind, that since this feature is not mentioned anywhere in Microsoft's docs it may not even be fully supported from Microsoft's side.
To submit comments, go to GitHub Discussions.