Sunday, January 19, 2014

Inconvenient MVC Web Optimization Framework and High CPU Utilization

While I am writing this post I am bit relaxed and listening to 'bollywood' music so don't wonder if you find anything off topic or typos :), lazy weekend!!

What was the issue?

Last week, while working with the Window azure cloud services we tried to stress test it and guess what we found? CPU spikes!!
Initially we thought that this might be something related to load on the server as it was a small VM instance but at the same time the results were unbelievable as number of concurrent requests to the server were comparatively less. This gave us the kick start and we started to drill down to the real issue.

The approach:

It all started with several meetings with few brainy people around as I was clueless, some even directly said man your application got memory licks..but then
Thanks to the Visual Studio profiling framework which helped me a-lot to drill down into real issue about this CPU spikes. You can read about how to use this feature and how it helps in my blog post here Profiling windows azure cloud services - http://passionatetechie.blogspot.in/2014/01/profiling-for-windows-azure-cloud.html

When I tried to stress test the service and tried to collect the profiled logs , found that there were these two methods which were taking almost more than 80% of CPU, wowwwwww!!
1. Scripts.Render
2. Styles.Render

 Note - I was using ASP.NET MVC web optimization nuget version 1.0

Irony is, these methods are from the web optimization framework of ASP.NET MVC which is suppose to improve the performance of your web application by bundling the java-scripts and css files. More info about bundling here - http://www.asp.net/mvc/tutorials/mvc-4/bundling-and-minification

All right then of course the straight way to remove these calls from application and try to stress application again and check results. we did exact same thing and this time application performed pretty well!! no CPU spikes!! man bingooo!!!!!!

But then why in my application the optimization framework was doing something like this? why ? why why and why.......? this question even appeared in my dreams!!
I started search around it but didn't find something helpful, I even started doubting my decision about removing those calls from the application unless I hit this post here
http://stackoverflow.com/questions/14210721/mvc-4-on-azure-scripts-render-bundles-jquery-slow and
http://stackoverflow.com/questions/12230246/azure-cache-preview-outputcache-high-cpu-slow

Here is why and it is purely based on what I understood

When you use the bundling (Web optimization framework) - server uses the caching.
But now as you are mentioning the setting in your web.config file that you are not using default caching of server anymore instead you want to use the Windows azure distributed cache.
Like this


<caching>
  <outputCache defaultProvider="DistributedCache">
    <providers>
      <add name="DistributedCache" 
type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, 
Microsoft.Web.DistributedCache" cacheName="default" dataCacheClientName="default" />
    </providers>
  </outputCache>
</caching>

So seems this web optimization framework doesn't understand this distributed cache and it simply ignores the caching part.
Due to this it tries to recompile your bundles every time with each request which causes higher CPU utilization.


I also had the distributed cache related settings in my application's web.config , so I decided to remove both from my application i.e. Bundling as well as distributed caching - it worked and CPU utilization never touched the boundary.


Takeaways:

In case you are facing the same issue of high CPU consumption on the web server in your ASP.NET MVC application or cloud service then
1. Run the VS Profiler locally and see whats the root cause of it
2. If you find the Optimization framework APIs are taking most of the CPU then try removing it and stress your application again to see the results.

Next step:

According to this link http://aspnetoptimization.codeplex.com/workitem/46 , its the discussion on codeplex on very same issue but If you go to the comments section then someone said that they have fixed it in the next version of nuget. (i.e. post version 1.0)
I havent tried using next version of this nuget but when I will do it I will update this post.

Hope this helps and stops wondering someone about why my CPU percentage is higher when using web optimization framework of MVC.







3 comments:

  1. So seems this web optimization framework doesn't understand this distributed cache and it simply ignores the caching part.

    I think that caching should be an integral part of any optimization framework in this case it is web.
    Also optimizing here means request serving optimization, so there is a high possibility of some configuration misinterpretation due to which, even when the cache present is useless.

    I this is not the case, then surely there would be bugs with the optimization framework.

    ReplyDelete