My whitepaper “How far can you go with SharePoint for ECM” has just been published. It brings together the information presented in a seminar this year at the Cimtech conference in Hatfield.

Follow this link and register to download the paper.

I’ve been working with my team to investigate some performance problems that one of our customers are having with their SharePoint website.

It’s a reasonably complex environment that has a couple of SharePoint farms, utilising web controls, custom http modules, web services and a raft of other things such as a number of backend systems.

Behold, ANTS!

To get some idea on what’s going on, we decided to use ANTS Profiler from RedGate, a product that shows you what’s happening with your environment and even the code that causes the issues.

The current release is 5.1 and you get to use it for 14 days. The cool thing is that if you need longer you can email them for keys that will extend the time by 14 days or try this key without the quotes – ‘i need more time’. Yep seriously, that is meant to work. The full version is £245, or £307 if you want support / upgrades.

Anyway, to get started you need to install ANTS on your server. ANTS can profile IIS applications, standalone .Net applications, web services, COM+ servers and of course SharePoint. One thing to note is that for profiling SharePoint I used version 5.2 that apparently has more SharePoint integration features. The SharePoint version is on the RedGate FTP site, so have a look at this forum post if you need that.

Actually, with regard to SharePoint, there’s a wee bit more configuration involved. This guide from RedGate specifies what you need to do to get it to work. It mentions using the Primary Site Collection Administrator as the profiling identity, however, Andrew Woodward states on his 21apps blog that the App Pool Identity should be used. I used the AppPool identity and that appears to work.

How do I use it?

Once you get ANTS installed and configured you simply open the profiler from the start menu. You’ll see something like this:

 

1. Two tabs are at the top, the main application settings page and the performance counters tab. The Perf counters tab allows you to add in additional performance counters. Beware though, this can degrade the performance of ANTS and skew your results. The main things to note however are on the Application Settings Tab.
2. In this case you want to select ASP.NET web application, enter the URL of your SharePoint site.
3. Ensure that you enter the correct port. If you leave it at 8013 it won’t work. Selecting the original port will ensure that it correctly picks up your site.

Once you click the ‘Start Profiling’ button, you’ll find that your website restarts. This ensures that you’ve got a clear run for profiling, but of your on a customers environment just be aware that it’ll bring down any other hosted IIS Sites that may be on the same box. Once the profiling runs you’ll see something like this:



Basically the big blue boxes with numbers correspond to the following:

1. Bookmark area – handy if you have a massive graph and want to see specific sections. Actually that’s one of the things you’ll notice about ANTS. You need to keep a note of what actions occur and in what order. It’s not that easy to determine if someone clicked on login.aspx or profile.aspx from the graph – you’ll just see a spike.

   You can actually raise a specific type of exception in your code and ANTS will pick it up. Details on how to raise ANTS exceptions and isolate ASP.Net pages can be found here:

   Isolating single ASP .NET pages in ANTS Profiler results

   I’d say that should only be used in pretty severe circumstances as you’ll have to redeploy production code again when finished. The best thing is to ensure that minimal people are on the environment and that all site operations are coordinated.

2. The Ectoplasm meter – Basically you’ll see spikes appearing here when something happens, such as a user clicking a button or submitting a page. You can use the mouse to select an area here and add a bookmark (point 1). One thing to note is the small thin lines in the ‘events area’ (between 1&2). Each line means that some sort of exception occurred. You can click on these and the profile will display where the error occurred in area 3.
3. The Call Tree – The call tree is the main area in which you’ll see what’s happening. The columns at the top are clickable, allowing you to sort by Method name, Time, Time with children and Hit Count. You can change the timing options by accessing the view menu or by hitting the F6, F7, F8 & F9 keys. A useful timing option is ‘Percentages (F6)’. This shows how much time is spent on a method compared with the entire page operation.

   The tree opens from the top downwards and you can drill into what composes a method by expanding the ‘+’ at the left of a tree node.

4. The Call Graph -  The call graph can be accessed by clicking the graph icon. What this does is provide you with a really great way of seeing where the method was called from and navigating through the call stack.

   You can basically click on the calling method and the screen redraws to show you what method called it, etc. The usefulness in this comes from the fact that you can traverse up the call stack to get to methods with names that actually make sense, rather than SPXmlDocCache.GetGlobalXmlDocumentCore(string PathXML, bool bFeature). To see what method called the method you’re looking at, you simply click it.

   Here’s what the call graph looks like

   

5. Change View – You can show the methods called in a number of ways. I’ve discussed the call tree and call graph, the final view is the method grid. Again it’s simply another view on the methods called, but in a tabular format. Sorting by Percentages / Milliseconds is especially useful in this view.
6. Display type – The display type allows you to tailor what methods ANTS shows you. You have options for Top down or Bottom up and both these can be further tweaked to show all methods or methods with source. I haven’t covered the source view yet, but yes ANTS will show you offending lines code if you have the PDB files available. I’ll cover this shortly.

   One thing to note is that if you don’t see any methods, try setting the display to all methods.

7. Source View – Yep, what we really want to see…. who’s fault it was that the code is running slow! The source view gives you a view on what code exists in a method. Clicking on a different method updates this view with the relevant code.

How do I view code?

Basically you need the PDB’s for your code. This means that you need to compile it in debug mode and copy these PDB’s to the server that’s hosting your code. The installation/configuration guide mentions that they need to go in the app_bin of of your IIS Site. I’ve had some issues with the PDB’s not being picked up and subsequently I’ve placed them in the Bin and GAC as well.

RedGate have an article on creating a ‘Global Debugging Symbols’ directory. I’ve not tried this so if you have any joy with it let me know.

The main thing to note is that even if you can’t see the code, ANTS gives you enough insight into the situation to make it worthwhile.

Conclusion

Overall I think ANTS Profiler is a great tool and it certainly shed’s light on code performance issues. The visual parts of the application are extremely impressive, especially as they make determining where a method is called from very straightforward.

Version 5.1 (and I believe 5.2) don’t yet have the facility to save ‘code views’ from your PDBs with the project, which is a shame. Hopefully the next version will address this.

There are alternatives of course, such as JetBrains profiler. I’ve not used this so I can’t comment on it’s effectiveness.

Thought I’d give some information around index files becoming corrupt on front end servers within a SharePoint farm.  This can sometimes happen due to a network or database failure while a crawl is in progress.  It will result in errors appearing in the Application Event Logs ressembling the following:

Event Type: Error
Event Source: Office Server Search
Event Category: Search service
Event ID: 10038
Date:                       8/8/2009
Time: 9:53:02 AM
User: N/A
Computer: WFE1
Description:
Query machine WFE1′ has been taken out of rotation due to this error: The
content index is corrupt.   0xc0041800.  It will be retried in 15 seconds.
Component: c9dba2c2-0077-46f7-b352-aeeddb6ac2fd

In the case where there are multiple front end servers in the farm, end users will remain unaffected since their search queries will be served by the other front end server, but this issue can cause crawls to hang and will log errors to the application logs every time a crawl is attempted.

This happened on one of our production environments recently, and the first thought was that we should reset the indexes and recrawl the content (which would have worked, but would have taken about 3 days to re-crawl everything and we wanted it working by Monday morning).  So in trying to avoid a recrawl, we did the following:

• Open the Operations tab of the Central Administration site
• Select Services on Servers and choose the offending WFE from the list
• Stop the Office Server Search Service on the WFE
• Start the service again, but give a new location for the index file
• The next time the crawl runs, the index will be propagated to the new location and bring the server back into rotation.

Hope this helps avoid many hours of re-crawling content.