During the last couple of weeks, users started noticing certain issues with an application that I develop on Microsoft Azure using PHP and Azure Web Apps. In general, the application is using an OAuth2 library to connect to a custom identity provider, the access token, refresh token and others are stored in $_SESSION. From the last sentence, you could have noticed two keywords – OAuth2 and refresh. The users started noticing an issue that after some time of working with the application (also the IdP was generating loads of useless token pairs), that they had to constantly relog. After turning on the logs, doing some diagnostics and trying to reproduce the issue, it was pretty clear – something fishy is going on with sessions.
What was actually happening was this – when the access token expires, the application attempts to renew the token using its refresh token. The issue was, that there could be parallel requests from the user (from AJAX calls for example) and the server would process them in parallel, which resulted in the following situation:
- REQUEST1: Inbound
- REQUEST2: Inbound
- REQUEST1: Access token is expired, attempt to renew
- REQUEST2: Access token is expired, attempt to renew
- REQUEST1: Got new access and refresh tokens, storing them into $_SESSION and fulfilling the request (at this point, the request is sent to client)
- REQUEST2: Token refresh failed, handle the failure, throw an error, etc.
From here, it is pretty obvious, REQUEST2 in parallel overwrites the token pair set into $_SESSION by REQUEST1. Let’s take a look at why this happens.
Session storage in Azure Web Apps
From the default configuration, using phpinfo() to list all the settings related to sessions, we can see, that the session.save_handler is set to wincache and that the session.save_path is set to D:\local\Temp. When we look into the directory using Kudu (https://yoursitename.scm.azurewebsites.net) we won’t see any session files in that directory – which turned me on a totally wrong path – I was actually trying to figure out where are the sessions being written to. However, when using scandir right in some PHP script, you will notice that some sessions are there.
The reason why you see the sessions in D:\local\Temp directly from the PHP script run in the browser is, that the Kudu (scm) is running in a different W3 Worker Process. Azure Web Apps is using a shared storage across instances which is located at D:\home however, each W3WP process has its own local, temporary storage.
To be honest, it took me a while to figure this fact out.
Going a little bit deeper – sandbox
In the text above, I mentioned that the local directory (D:\local) is different for each W3WP. This is set by default to separate the SCM and actual web application. When I write set, it means that this can be actually changed! It is not mentioned much at many places (only GitHub documentation), but there is an option for it.
When you set an application setting or the .deployment file to contain WEBSITE_DISABLE_SCM_SEPARATION=true. It is not going to sandbox the SCM process away from your application.
With that option in place, you will be able to see the content of D:\local from the web application in Kudu including Temp directory and its contents.
Last important thing is that this option should be mostly used for debugging and not for production (in production, you want as much isolation as you can get!).
Concurrent writing into sessions
There are many different ways on how to handle concurrent writing – you could use locks, semaphores and more. Speaking of PHP, it prevents concurrent session writing by locking onto specific session file and whenever you are done with writing, you can call session_write_close to remove the lock and let other request with same session id to go through.
Tip: Session locks and concurrency are very nicely explained in this article.
Obviously, in my case, this wasn’t happening. The reason is, that with the default configuration, PHP on Azure Web Apps is set to use WinCache handler for sessions, despite the fact that it really works well, it kind of silently leaves out the session locking part.
After figuring this fact out, all I had to do was to switch session.save_handler to files, restart the website (for the changes in settings.ini to take effect immediately) and everything was working again as expected.
WinCache sessions vs file sessions
So what’s this WinCache and what is it good for? WinCache in general is a cache which makes PHP on Windows faster (read more about it here). It integrates with the opcode, user cache, file cache and also sessions. Major enhancement to sessions is that it stores the sessions in shared memory (and uses filesystem to persist the data during Application Pool recycle in IIS) which allows for faster access to session data.
As opposed to regular file session cache in PHP, it doesn’t implement session locking for writes, so you may ned up with various issues like I ran into.
To wrap it up, I simply switched back to regular files in session.save_handler configuration and everything is now working as expected with session locks.