Redgate's ANTS Memory Profiler is a fantastic tool to help track potential memory leaks. But, it is important to have realistic expectations when getting started using the tool. When used properly, the tool can be very helpful in tracking down memory leaks. By no means is it a silver bullet. There is a learning curve with the tool. It will do all the heavy lifting of profiling all the items in memory, but it will not point at a specific piece of code and tell you that is where the memory leak is happening.
A few weeks ago I was tasked with tracking down a potential memory leak for one of the applications at my new job. I say potential memory leak because this application has been rock solid for many years. There was some doubt of a memory leak, the only way to know for sure is to use on of the many memory profilers out there. ANTS Memory Profiler is among the best of the best in that field. One of the perks of being a "Friend of Redgate" is it includes free licenses to their various products. I downloaded Redgate's ANTS Memory Profiler and started digging into the problem.
It has been a while since I have had the chance to use this tool, or a memory profiler in general. The next series of articles will cover some of the lessons I had to relearn in order to use the tool effectively. This article will cover some of the fundamentals in memory profilers in general. Future articles will dive into the tool itself.
#1 - Get the use case of the potential memory leak
This one should seem fairly obvious, but I've been party to a number of times where the person investigating the memory leak didn't bother to do this step. Without it, they were just randomly clicking around the application.
Memory leak reports come in many forms. The two most common ones are instances reported by users or a rogue process consuming all the memory on a server causing alerts to fire. Server memory leaks are an extra joy to debug, it requires tooling such as NewRelic to find out what calls were being made on the server.
In the case where I was asked to help track down the memory leak, it was reported by users of the application. Having users report the issue can be very helpful, more people having a problem indicates a trend which can help track down the problem.
Some of the common questions I ask when a user reports a memory leak are:
- How long were you using the application before you noticed an issue? A couple of hours? A couple of days?
- When you noticed the memory leak what were you doing in the application?
- Was there anything out of the ordinary you were doing moments before you noticed the memory leak?
- How do you typically use the application? In the period up to and after the memory, issue were you doing what you would consider normal usage?
- How often does it happen?
Once you have that use case in place put together a test plan.
#2 Know the application or talk to submit matter experts
In the application I was looking at recently I noticed there were a lot of references to resource files. I thought that was odd, so I asked the development manager. He said each client of the application gets their own resource file, this is the application is sold as customizable to the client's liking. Not much I can do about that. So I moved on to other items.
Because of the repetitive manual testing, tracking down memory leaks is often assigned to more junior members of the development teams. Or newer members of a team. If you are on the side doing the assigning make yourself available to answer any and all questions about the application. If you are on the side where the assigning is happening to you pretend you are five years old and keep on asking why to the senior members of the team. Maybe the right question will be asked to trigger something.
#3 - Be Methodical
Most memory leaks are very hard to discover. They would have shown up almost right away during testing if they were apparent. Memory profilers are not magic tools. You don't just fire them up and they will instantly point at the memory leak. Prepare yourself mentally for the long haul. I've looked at many memory leaks in the past with the expectation it would be solved in a couple of hours. That is almost never the case. It takes a long methodical process to find the cause of the memory leak. Don't be surprised if it takes days to uncover.
Have a good and repeatable test plan. And don't be afraid to alter the plan as more information comes to light. A couple of years ago I was tasked with finding a memory leak in a Single Page Application where it would only show up after several hours of usage when the user was using IE 11. When I asked the user how they were using the application they told me they just left it running on a particular page (it was a dashboard that auto refreshed). My plan was to take a memory snapshot at set intervals.
- Load dashboard and take baseline snapshot.
- Take snapshot after 10 minutes.
- Take snapshot after 30 minutes.
- Take snapshot after 60 minutes.
- Take snapshot after 90 minutes.
- Take snapshot after two hours.
- Take snapshot after three hours.
- Take snapshot after four hours.
When you believe you have solved the problem then repeat the same test. It is super fun to do. Just like a root canal is fun. But it is the only way to know for certain if the fix took.
In this particular case, I was lucky because the application auto refreshed. Having to manually click through the application for that length of time would have driven me mad. If anything, it sure would have been a good case for automation!
#4 Have a basic understanding of how memory is allocated and garbage collected
I love that .NET handles all the memory allocation and garbage collection automatically. I didn't even bother to learn the basics of memory for a long time. I blissfully kept my head in the sand the first time I was assigned a task of tracking down a memory leak. I got a crash course in .NET memory management. It is not a very easy topic to wrap your head around, it takes some time to learn everything and as a result, it is something you should avoid having to do in the middle of a crisis.
Key topics to learn:
- How memory is allocated in .NET
- The difference between the heap and the stack
- How items promoted from generation 0 to generation 1 to generation 2
- Why items are moved to the large object heap and how that can cause an issue
- Differences in types of Garbage Collection
- How .DLLs are loaded during runtime
#5 Look For Common Resource Hogs
Anytime I go into an application looking for a memory leak I first look for
- Collections being stored in memory long term, in a static variable or a class which is static. Dictionaries are big time offenders because .NET has to keep a hash table in memory. If the dictionary grows large enough it can be thrown onto the heap. For a long running application, such as a Windows Service or a Win32 application that is a real possibility.
- Anything using streams or byte arrays, these can easily grow out of control if not used correctly.
- Anything not using a using statement on an IDisposable object. The more common cases include SqlConnection, SqlCommand, DataReader, and DataTable. You might be thinking, in my application, I only use EntityFramework or Dapper. Well, a lot of applications were written before the time where an ORM was a common practice and a lot of hand-written ADO.NET code was written.
If you see a lot of these in the memory profiler it might be something worth looking into.
#6 Memory Is Managed by .NET, Not By You
.NET manages the memory, that is one of the big perks of using that framework. The garbage collector was written by people much, much smarter than you or I. What I am trying to say is if you find yourself in a situation where you are looking into ways to trick the garbage collector or change how memory is allocated in .NET then stop what you are doing. Nothing good will come from that.
Conclusion
This article was written to provide a foundation for future articles. In my next article, I will walk through the ANTS memory profiler.
