How to troubleshoot memory related issues

In this post I wil how to troubleshoot high CPU usage issues caused by w3wp.exe. Recently, I received a call from Client about the high CPU usage by w3wp.exe. w3wp.exe was consuming 90% of CPU usage. After bit of a research, I came to know that to troubleshoot these kind of issues we need to take a memory dump of the system at the time when issue occurs. Once we have the memory dumps, there are some tools that can be used to do root cause analysis. This troubleshoot is a two-step process.

Step 1 – Gather Data

The easiest way to collect user-mode process dumps when a high CPU condition occurs is to use Debug Diagnostics 1.2, or DebugDiag. You can download DebugDiag from the following URL.

http://www.microsoft.com/download/en/details.aspx?id=26798

Install DebugDiag 1.2 on your server and run it. (You’ll find it on the Start menu after installation.) When you run DebugDiag, it will display the Select Rule Type dialog. Follow these steps to create a crash rule for your application pool.

1. Select Performance and click Next.
2. Select Performance Counters and click Next.
3. Click Add Perf Triggers.
4. Expand the Processor (not the Process) object and select % Processor Time. Note that if you are on Windows Server 2008 R2 and you have more than 64 processors, please choose the Processor Information object instead of the Processor object.)
5. In the list of instances, select _Total.
6. Click Add and then click OK.
7. Select the newly added trigger and click Edit Thresholds as shown in Figure below.
8. Select Above in the dropdown.
9. Change the threshold to 80.
10. Enter 20 for the number of seconds. (You can adjust this value if needed, but be careful not to specify a small number of seconds in order to prevent false triggers.)
11. Click OK.
12. Click Next.
13. Click Add Dump Target.
14. Select Web Application Pool from the dropdown.
15. Select your application pool from the list of app pools.
16. Click OK.
17. Click Next.
18. Click Next again.
19. Enter a name for your rule if you wish and make note of the location where the dumps will be saved. You can change this location if desired.
20. Click Next.
21. Select Activate the Rule Now and click Finish.

TIP

You can create dumps of multiple application pools by adding multiple dump targets using the same technique used in steps 13-15.

This rule will create 11 dump files. The first 10 will be “mini dumps” which will be fairly small in size. The final dump will be a dump with full memory, and that dumps will be much larger.

Step 2 – Data Analysis

DebugDiag has the ability to recognize many problems by doing an automated dump analysis. For this particular issue, DebugDiag’s Performance Analyzers are well-suited to helping to identify the root cause of the high CPU issue. To use the analyzer, follow these steps

1. Select the Advanced Analysis tab in DebugDiag.
2. Select the Performance Analyzers.
3. Click Add Data Files.
4. Browser to the location where the dumps were created. By default, this will be a subfolder of the C:\Program Files\DebugDiag\Logs folder.
5. Select one of the dumps and then press Ctrl+A to select all of the dumps in that folder.
6. Click Open.
7. Click Start Analysis.

DebugDiag will take a few minutes to parse through the dumps and provide an analysis. When it has completed the analysis, you will see a page similar to that shown in Figure below.

Notice that the top of the report tells you that high CPU was detected. In the right column, you’ll see recommendations which include a link to the top 7 threads by average CPU time. Click that link and you’ll see information about what those top CPU consumers were doing. Figure 7 shows what those threads are doing in my application.

I can tell from this analysis that the default.aspx page  is running. If I look further down the call stack (at the bottom of the page), I can see that this thread is doing string concatenation. (Notice the call to System.String.Concat on the call stack.) If I analyze the other top CPU threads, I see the same pattern.

The next step is to review the Page_Load event in the default.aspx page. Have a look at the code in the page_Load event and see what can be improved.

The post How to troubleshoot memory related issues appeared first on Manvir's SharePoint Blog.