Tackling memory leak

In academia, there is no difference between academia & the real world.

In the real world, there is.

Nassim Nicholas Taleb

This post covers the real world case study on tackling memory leak.

Investigation strategy

Memory leak occurs if useless resources are not removed from memory – there has to be something rooting old data. Investigation plan:

1. Collect memory snapshot once memory usage is beyond the expected range
2. Collect another one in a minute or so
3. Compare heap statistics: locate growing/weird types
4. Find GC roots for these objects
5. Determine construction mechanisms for these instances

Collecting memory snapshots

A few memory snapshots have been collected via Task Manager:

Snapshot can be collected automatically once memory is higher than threshold.

Any anomalies in heap statistics?

Both snapshots contain a huge number of GlassMapper objects that contradicts the module usage pattern (create many short-lived objects):

• 

• 

We could either find !gcroots (who holds object in memory) for suspicious classes that have much higher count than expected or look for unexpectedly huge objects in the heap.

A few IDisposable[] arrays take as much space as 100K objects?! That is over 2200 elements in average (or ~18KB) inside array. There are over 700 huge arrays each having over two thousands disposable elements each, sounds as a code smell.

Finding bigger than average arrays with size over 20K bytes (via min param):

Two million elements long array (disposable GlassMapper objects inside) is rooted to ServiceProvider:

Same pattern is met in all the captured snapshots.

Hypothesis

GlassMapper disposable objects are resolved from global DI scope, hence getting disposed when application shuts down. Objects are stacked in memory until then.

Alternatively disposed scope is used to keep on resolving objects.

Who creates objects that are never disposed?

Although leaking classes can be modified to save birth call stack & ServiceProvider used to create them, not owned types cannot be modified to add that code.

Nevertheless, behavior could be simulated on a test disposable class and Sitecore DI mechanism.

Demo code

The disposable dependency records its birth call stack and capture the ‘parent’ service provider:

using System;
using System.Threading;
using Sitecore.Abstractions;
using Sitecore.Diagnostics;
using Sitecore.Pipelines;

namespace Demo
{
    /// <summary>
    /// Registered in DI with transient scope.
    /// <para>Dispose is to be called upon scope end.</para>
    /// </summary>
    public class DisposableDependency : IDisposable
    {
        private readonly BaseLog _log;
        private readonly IServiceProvider _parent;
        private static int _instancesCount;
        private readonly string _birthCertificate;

        public int Number { get; }

        public DisposableDependency(BaseLog log, IServiceProvider parent)
        {
            Assert.ArgumentNotNull(log, nameof(log));
            _log = log;
            _parent = parent;
            Number = Interlocked.Increment(ref _instancesCount);
            _log.Warn($"Dummy processor was born: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
            _birthCertificate = Environment.StackTrace;
        }

        public void Process(PipelineArgs args)
        {
            _log.Warn($"Dummy processor was executed: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
        }

        void IDisposable.Dispose() => _log.Warn($"Dummy processor died: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
    }
}

Injection to types created by Factory

Sitecore creates objects via DI if resolve=’true’ attribute set in config node.

New instance is created each time if reusable=’false’ attribute is set for processor:

<?xml version="1.0"?>

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
    <sitecore>
        <services>
            <!-- Register disposable service -->
            <register serviceType="Demo.DisposableDependency, Demo" implementationType="Demo.DisposableDependency, Demo" />
        </services>

        <pipelines>
            <initialize>
            <!-- Register controller route to test MVC constructor injection -->
                <processor type="Demo.RegisterRoute, Demo"/>
            </initialize>

            <disposableSample>
            <!-- Register processor to be created every time -->
                <processor type="Demo.DisposableDependency, Demo" resolve="true" reusable="false" />
            </disposableSample>
        </pipelines>
    </sitecore>
</configuration>

Define the disposableSample pipeline with non-reusable DisposableDependency so that a new instance is created for each call. Invoke it via dummy page:

<%@ Page Language="C#" AutoEventWireup="true" Inherits="Demo.PipelineCall" %>

using System;
using Sitecore.Abstractions;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;
using Sitecore.Pipelines;

namespace Demo
{
    [AllowDependencyInjection]
    public class PipelineCall : System.Web.UI.Page
    {
        private readonly BaseCorePipelineManager _pipelineManager;

        protected PipelineCall()
        {
        }

        public PipelineCall(BaseCorePipelineManager pipelineManager)
        {
            Assert.ArgumentNotNull(pipelineManager, nameof(pipelineManager));
            _pipelineManager = pipelineManager;
        }

        protected void Page_Load(object sender, EventArgs e)
        {
            var pipeline = _pipelineManager.GetPipeline("disposableSample", string.Empty);
            pipeline.Run(new PipelineArgs());
        }
    }
}

Injection to WebForms and UserControls

ASP.NET ConstructorInjection.aspx form:

<%@ Page Language="C#" AutoEventWireup="true" Inherits="Demo.ConstructorInjection" %>

using System;
using System.Web;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;

namespace Demo
{
    [AllowDependencyInjection]
    public partial class ConstructorInjection : System.Web.UI.Page
    {
        private readonly DisposableDependency _dependency;

        protected ConstructorInjection()
        {
        }

        public ConstructorInjection(DisposableDependency dependency)
        {
            Assert.ArgumentNotNull(dependency, nameof(dependency));
            _dependency = dependency;
        }

        protected void Page_Load(object sender, EventArgs e)
        {
            HttpContext.Current.Response.Write($"<h2>I was called {_dependency.Number} </h2>");
        }
    }
}

Injection and MVC

Controller that needs disposable dependency:

using System.Web.Mvc;
using Demo;
using Kerzner.Foundation.DependencyInjection;

namespace Kerzner.Project.BrandWebsites.Pipelines.ScScopeVerification
{
    [Controller]
    public class DummyController : Controller
    {
        private readonly DisposableDependency _dependency;

        public DummyController(DisposableDependency dependency)
        {
            _dependency = dependency;
        }

        public ActionResult Index()
        {
            return Content($"<h2>called {_dependency.Number} <h2/>");
        }
    }
}

Controller registration:

using System.Web.Mvc;
using System.Web.Routing;
using Sitecore.Pipelines;

namespace Demo
{
    public class RegisterRoute
    {
        public void Process(PipelineArgs args)
        {
            RouteTable.Routes.MapRoute(
                "DisposableDependency",
                "DummyController",
                new { controller = "Dummy", action = "Index" });
        }
    }
}

Test results

Non-disposed dependencies are created by Factory

Sitecore DefaultFactory (registered as singleton) captures scoped ServiceProvider. Any object is created from that service provider reuses that scope instead of one valid for request. Birth certificates proof objects created by Factory thereby leaking memory.

ASP.NET pages do not dispose some dependencies as well

AutowiredPageHandlerFactory creates ASPX pages with the respect to proper scope from the first sight. Nevertheless, it does not work as intended due to handler factory cache inside HttpApplication.

Even though correct service provider scope is used at first, it would still be re-used for further requests processed by the instance of HttpApplication.

Conclusion

The memory leak mystery was resolved thanks to a few memory snapshots. Once the cause is found, code adjustments could be done:

• Do not resolve disposable objects from DI, prefer explicit creation (in case possible) to favor explicit dispose call
• Consume per-request scope from Sitecore.DependencyInjection.SitecorePerRequestScopeModule.GetScope to resolve dependencies from request-bound scope so that disposed is called on EndRequest
• Sitecore framework improvement to keep in mind potential nature of disposable dependencies

The behavior described in the article has been reported to Sitecore Support and accepted as 331542 bug report.