One of the most commonly used patterns in software development is Caching. It’s a simple, but a very effective concept. The idea is to reuse operation results. When performing a heavy operation, we will save the result in our cache container. The next time that we need that result, we will pull it from the cache container, instead of performing the heavy operation again.

For example, to get a person’s Avatar you might need a trip to the database. Instead of performing that trip every time, we will save that Avatar in the cache, pulling it from memory every time you need it.

Caching works great for data that changes infrequently. Or even better, never changes. Data that constantly changes, like the current machine’s time shouldn’t be cached or you will get wrong results.

In-process Cache, Persistant in-process Cache, and Distributed Cache

There are 3 types of caches:

  • In-Memory Cache is used for when you want to implement cache in a single process. When the process dies, the cache dies with it. If you’re running the same process on several servers, you will have a separate cache for each server.
  • Persistent in-process Cache is when you back up your cache outside of process memory. It might be in a file, or in a database. This is more difficult, but if your process is restarted, the cache is not lost. Best used when getting the cached item is expensive, and your process tends to restart a lot.
  • Distributed Cache is when you want to have shared cache for several machines. Usually, it will be several servers. With a distributed cache, it is stored in an external service. This means if one server saved a cache item, other servers can use it as well. Services like Redis are great for this.

We’re going to talk just about in-process cache.

Naive Implementation

Let’s create a very simple cache implementation in C#:

public class NaiveCache<titem>
{
    Dictionary<object titem=""> _cache = new Dictionary<object titem="">();

    public TItem GetOrCreate(object key, Func<titem> createItem)
    {
        if (!_cache.ContainsKey(key))
        {
            _cache[key] = createItem();
        }
        return _cache[key];
    }
}</titem></object></object></titem>

Usage:

var _avatarCache = new NaiveCache<byte>();
// ...
var myAvatar = _avatarCache.GetOrCreate(userId, () => _database.GetAvatar(userId));</byte>

This simple code solves a crucial problem. To get a user’s avatar, only the first request will actually perform a trip to the database. The avatar data (byte[]) is then saved in process memory. All following requests for the avatar will be pulled from memory, saving time and resources.

But, as most things in programming, nothing is so simple. The above solution is not good for a number of reasons. For one thing, this implementation is not thread-safe. Exceptions can occur when used from multiple threads. Besides that, cached items will stay in memory forever, which is actually very bad.

Here’s why we should be removing items from Cache:

  1. Cache can take up a lot of memory, eventually leading to an out-of-memory exceptions and crashes.
  2. High memory consumption can lead to GC Pressure (aka Memory Pressure). In this state, the garbage collector works more than it should, hurting performance.
  3. Cache might need to be refreshed if the data changes. Our caching infrastructure should support that ability.

To handle these problems, cache frameworks have Eviction policies (aka Removal policies). These are rules to have items removed from cache according to some logic. Common eviction policies are:

  • Absolute Expiration policy will remove an item from cache after a fixed amount of time, no matter what.
  • Sliding Expiration policy will remove an item from cache if it wasn’t accessed in a fixed amound of time. So if I set the expiration to 1 minute, the item will keep staying in cache as long as I use it every 30 seconds. Once I don’t use it for longer than a minute, the item is evicted.
  • Size Limit policy will limit the cache memory size.

Now that we know what we need, let’s continue on to better solutions.

Better Solutions

To my great dismay as a blogger, Microsoft already created a wonderful cache implementation. This deprived me the pleasure of creating a similar implementation myself, but at least I have less work writing this blog post.

I’ll show you Microsoft’s solution, how to effectively use it, and then how to improve it in some scenarios.

System.Runtime.Caching/MemoryCache vs Microsoft.Extensions.Caching.Memory

Microsoft has 2 solutions 2 different NuGet packages for caching. Both are great. As per Microsoft’s recommendation , prefer using Microsoft.Extensions.Caching.Memory because it integrates better with Asp. NET Core. It can be easily injected into Asp .NET Core’s dependency injection mechanism.

Here’s a basic example with Microsoft.Extensions.Caching.Memory:

public class SimpleMemoryCache<titem>
{
    private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions());

    public TItem GetOrCreate(object key, Func<titem> createItem)
    {
        TItem cacheEntry;
        if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
        {
            // Key not in cache, so get data.
            cacheEntry = createItem();
            
            // Save data in cache.
            _cache.Set(key, cacheEntry);
        }
        return cacheEntry;
    }
}</titem></titem>

Usage:

var _avatarCache = new SimpleMemoryCache<byte>();
// ...
var myAvatar = _avatarCache.GetOrCreate(userId, () => _database.GetAvatar(userId));</byte>

This is very similar to my own NaiveCache, so what changed? Well, for one thing, this is a thread-safe implementation. You can safely call this from multiple threads at once.

The second thing thing is the MemoryCache allows for all the eviction policies we talked about before. Here’s an example:

IMemoryCache with eviction policies:

public class MemoryCacheWithPolicy<titem>
{
    private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions()
    {
        SizeLimit = 1024
    });

    public TItem GetOrCreate(object key, Func<titem> createItem)
    {
        TItem cacheEntry;
        if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
        {
            // Key not in cache, so get data.
            cacheEntry = createItem();

            var cacheEntryOptions = new MemoryCacheEntryOptions()
             .SetSize(1)//Size amount
             //Priority on removing when reaching size limit (memory pressure)
                .SetPriority(CacheItemPriority.High)
                // Keep in cache for this time, reset time if accessed.
                .SetSlidingExpiration(TimeSpan.FromSeconds(2))
                // Remove from cache after this time, regardless of sliding expiration
                .SetAbsoluteExpiration(TimeSpan.FromSeconds(10));

            // Save data in cache.
            _cache.Set(key, cacheEntry, cacheEntryOptions);
        }
        return cacheEntry;
    }
}</titem></titem>

Let’s analyze the new additions:

  1. SizeLimit was added in MemoryCacheOptions. This adds a size-based policy to our cache container. Size doesn’t have a unit. Instead, we need to set the size amount on each cache entry. In this case, we set the amount to 1 each time with SetSize(1). This means that the cache is limited to 1024 items.
  2. When we reach the size limit, which cache item should be removed? You can actually set priority with .SetPriority(CacheItemPriority.High). The levels are Low, Normal, High, and NeverRemove.
  3. SetSlidingExpiration(TimeSpan.FromSeconds(2)) was added, which sets sliding expiration to 2 seconds. That means if an item was not accessed in over 2 seconds it will be removed.
  4. SetAbsoluteExpiration(TimeSpan.FromSeconds(10)) was added, which sets absolute expiration to 10 seconds. This means than the item will be evicted within 10 seconds if it wasn’t already.

In addition to the options in the example, you can also set a RegisterPostEvictionCallback delegate, which will be called when an item is evicted.

That’s a pretty comprehensive feature set. It makes you wonder if there’s even anything else to add. There are actually a couple of things.

Problems and Missing features

There are a couple of important missing pieces in this implementation.

  1. While you can set the size-limit, the caching doesn’t actually monitor gc pressure. If we did monitor it, we could tighten policies when the pressure is high, and loosen up policies when the pressure is low.
  2. When requesting the same item with multiple threads at the same time, the requests don’t wait for the first one to finish. The item will be created multiple times. For example, let’s say we are caching the Avatar, and getting an avatar from the database takes 10 seconds. If we request an avatar 2 seconds after the first request, it will check if the avatar is cached (it isn’t yet), and start another trip to the database.

As for the first problem of gc pressure: It’s possible to monitor GC pressure with several techniques and heuristics. This blog post is not about that, but you can read my article Find, Fix, and Avoid Memory Leaks in C# .NET: 8 Best Practices to learn of some helpful methods.

The second problem is easier to solve. In fact, here’s an implementation of MemoryCache that solves it entirely:

public class WaitToFinishMemoryCache<titem>
{
    private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions());
    private ConcurrentDictionary<object semaphoreslim=""> _locks = new ConcurrentDictionary<object semaphoreslim="">();

    public async Task<titem> GetOrCreate(object key, Func<task>> createItem)
    {
        TItem cacheEntry;

        if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
        {
            SemaphoreSlim mylock = _locks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));

            await mylock.WaitAsync();
            try
            {
                if (!_cache.TryGetValue(key, out cacheEntry))
                {
                    // Key not in cache, so get data.
                    cacheEntry = await createItem();
                    _cache.Set(key, cacheEntry);
                }
            }
            finally
            {
                mylock.Release();
            }
        }
        return cacheEntry;
    }
}</task></titem></object></object></titem>

Usage:

var _avatarCache = new WaitToFinishMemoryCache<byte>();
// ...
var myAvatar = 
 await _avatarCache.GetOrCreate(userId, async () => await _database.GetAvatar(userId));</byte>

With this, when trying to get an item, if the same item is in the middle of being created by another thread, you will wait for the other to finish first. Then, you will get the already cached item created by the other thread.

Explanation of the code

This implementation locks the creation of an item. The lock is specific to the key. For example, if we’re waiting to get Alex’s Avatar, we can still get cached values of John or Sarah on another thread.

The dictionary _locks stores all the locks. Regular locks don’t work with async/await, so we need to use SemaphoreSlim .

There are 2 checks to see if the value is already cached if (!_cache.TryGetValue(key, out cacheEntry)). The one inside the lock is the one that ensures there’s a single creation. The one outside of the lock is for optimization.

When to use WaitToFinishMemoryCache

This implementation obviously has some overhead. Let’s consider when it’s even necessary.

Use WaitToFinishMemoryCache when:

  • When the creation time of an item has some sort of cost, and you want to minimize creations as much as possible.
  • When the creation time of an item is very long.
  • When the creation of an item has to be ensured to be done once per key.

Don’t use WaitToFinishMemoryCache when:

  • There’s no danger of multiple threads accessing the same cache item.
  • You don’t mind creating the item more than once. For example, if one extra trip to the database won’t change much.

Summary

Caching is a very powerful pattern. It’s also dangerous and has its own complexities. Cache too much and you can cause GC pressure. Cache too little and you can cause performance issues. Then, there’s distributed caching, which is a whole new world to explore. That’s software development for you, always something new to learn.

I hope you enjoyed this post. If you’re interested in memory management, my next article is going to be about the dangers of GC pressure and techniques to prevent it, so keep following . Happy coding.

M ↓   Markdown
?
Anonymous
0 points
6 years ago

Anyone who is interested in boosting ASP.NET application performance can download this whitepaper to get to know about 4 ways to optimize performance and remove bottlenecks.
http://www.alachisoft.com/r...

?
Anonymous
0 points
6 years ago

NCache is a great product!

?
Anonymous
0 points
6 years ago

In your WaitToFinishMemoryCache implemetation you are using below code,
SemaphoreSlim mylock = _locks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));

If ConcurrentDictionar.GetOrAdd simultaneously on different threads, valueFactory may be called multiple times, would not cause any issues ?

?
Anonymous
0 points
6 years ago

Hi,
The code ensures it will not be called multiple times on different threads for the same key. It can be called multiple times for different keys, in which case it depends on the delegate content.

?
Anonymous
0 points
6 years ago

Michael, GetOrAdd it not thread-safe, so madhu is right. You can make it thread-safe using Lazy. https://andrewlock.net/maki...

?
Anonymous
0 points
6 years ago

Thanks Luis, I read it now.
From my understanding, using Lazy ensures the delegate "new SemaphoreSlim(1, 1)" runs only once. But, even with current implementation it will always return the same SemaphoreSlim instance for all threads, which is what matters. So it does't matter if the lock creation delegate runs more than once. It does matter if the value creation delegate runs more than once, but this is not the case.
So you're right that GetOrAdd can run the delegate more than once, but it's OK for our purposes.

?
Anonymous
0 points
6 years ago

Hi Michael, I agree with you. In this case it's not worth to use lazy. I've pointed out that GetOrAdd is not thread-safe because not all are aware of it and it can end with unexpected side effects.

?
Anonymous
0 points
6 years ago

Luis, you've actually solved an issue for me as to why the Microsoft MemoryCache is not thread-safe even though it uses ConcurrentDictionary internally, so Thanks!

?
Anonymous
0 points
6 years ago

Just an observation.... In the past 10 years I have been brought in to address "performance issues" at many clients. In three recent occasions, the client teams had worked hard to "cache and optimize" object lifetimes. There code looked well thought out, and they could even produce graphs that showed they helped performance....

My firm ripped all of that out, and minimized the implementations to "create, use, let-go" with the focus on the shortest object lifecycle possible. Yes, these even meant removing caching of semi-fixed lookup data from the database! The results was that overall application performance improved!

In summary, Michael' comment of "Caching is a very powerful pattern. It's also dangerous and has its own complexities." and the results can be quite counter intuitive.

?
Anonymous
0 points
6 years ago

Cool story David. That just goes to show how over optimization or incorrect optimization can actually make things worst.

?
Anonymous
0 points
6 years ago

For async operation better use https://docs.microsoft.com/... which now include memory realization.

?
Anonymous
0 points
6 years ago

Thanks for the tip Denis

?
Anonymous
0 points
6 years ago

Great article! One more thing is missing from your implementation: locks never removed from the _locks ConcurrentDictionary, so it can grow unlimited and can cause out of memory exception.

?
Anonymous
0 points
6 years ago

Thanks and nice catch!

?
Anonymous
0 points
6 years ago

Was this updated to prevent unlimited growth.. didn't see the article before this so I'm not aware of changes.

?
Anonymous
0 points
6 years ago

Nope, this is standalone

?
Anonymous
0 points
6 years ago

That was the first thing that came to mind when I saw that extra ConcurrentDictionary in the code. Funny since the article explicitly emphasizes the need to remove objects from cache and not have them linger around forever :-)

?
Anonymous
0 points
6 years ago

Good post! I was looking for a memory cache implementation to attach it in a repository pattern. The truth is that seeing the code seems an elegant way to solve the problem of concurrence in memory access. I have seen in the N-Ware commentary the problem of not freeing the memory of the SemaphoreSlim dictionary _lock, and investigating a bit I found a CallBack (PostEvictionCallBack) launched from the MemoryCache itself when a record is deleted. In case it helps.

Again nice post, and very usefull! :)

?
Anonymous
0 points
6 years ago

I agree that callback can be a good place for cleanup. I'll update the article when I have some time.
And thanks!

?
Anonymous
0 points
6 years ago

Ideally, we should avoid creating our own GetOrCreate method when IMemoryCache already comes with an extension method called... GetOrCreate! Granted, there are some unresolved issues with that (https://github.com/aspnet/E...), although they are partially contradicted in a similar discussion (https://github.com/aspnet/C...). In any case, I expect the issues will be resolved soon. Hopefully they will make the Core 3.0 release.

Depending on how the issues are resolved, the resulting behavior might be like ConcurrentDictionary: a race condition may lead to multiple values being created, but only one value is ever stored and returned. If it is truly undesirable to create a value multiple times (and often it does not really matter, as a one-time cost!), we can simply wrap the cached value in a Lazy (with the right LazyThreadSafetyMode), and then only one value will ever be created.

?
Anonymous
0 points
6 years ago

Timo I agree and hope this will be resolved within the library. Thanks for the links

?
Anonymous
0 points
5 years ago

Why class WaitToFinishMemoryCache is not static?

?
Anonymous
0 points
5 years ago

When should the cache be used as a singleton or scoped DI instance and what are the factors to consider? If creating the cache as a singleton if its used with any data layer that is scoped that would not work but is there performance issues if a concurrent dictionary is scoped where it would recreate the cache in memory each time?

?
Anonymous
0 points
5 years ago

You can decouple your memory cache from your model and still have implemented an async method.
Here is your modified code:
public class MemoryCacheWithPolicy<TParameter, TResult>
{
static ILogger _nlogger = new AppLogger().Logger;
private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions()
{
SizeLimit = 1024
});

`` /// <summary>
/// Gets or creates a new memory cache record for a main data
/// along with parameter data that is assocciated with main main.
/// </summary>
/// <param name="key">Main data cache memory key.</param>
/// <param name="param">Parameter model that assocciated to main model (request result).</param>
/// <param name="createCacheData">A delegate to create a new main data to cache.</param>
/// <returns></returns>
public async Task<TResult> GetOrCreate(object key, TParameter param, Func<TParameter, TResult> createCacheData)
{
// this key is used for param cache memory.
var paramKey = key + nameof(param);

    if (!_cache.TryGetValue(key, out TResult cacheEntry))  
    {  
        // key is not in the cache, create data through the delegate.  
        cacheEntry = createCacheData(param);  
        createMemoryCache(key, cacheEntry, paramKey, param);  

        _nlogger.Warn(" cache is created.");  
    }  
    else  
    {  
        // data is chached so far..., check if param model is same (or changed)?  
        if(!_cache.TryGetValue(paramKey, out TParameter cacheParam))  
        {  
            //exception: this case should not happened!  
        }  

        if (!cacheParam.Equals(param))  
        {  
            // request param is changed, create data through the delegate.  
            cacheEntry = createCacheData(param);  
            createMemoryCache(key, cacheEntry, paramKey, param);  
            _nlogger.Warn(" cache is re-created (param model has been changed).");  
        }  
        else  
        {  
            _nlogger.Trace(" cache is used.");  
        }  

    }  
    return await Task.FromResult&lt;TResult&gt;(cacheEntry);  
}  
MemoryCacheEntryOptions createMemoryCacheEntryOptions()  
{  
    // Cache data within 2 seconds requests if the request model is   
    // not changed! Request new result after 5 seconds regardless...  
    return new MemoryCacheEntryOptions()  

        //Size amount  
        .SetSize(1024)  

        // Priority on removing when reaching size limit (memory pressure)  
        .SetPriority(CacheItemPriority.High)  

        // Keep in cache for this amount of time, reset it if accessed.  
        .SetSlidingExpiration(TimeSpan.FromSeconds(2))  

        // Remove from cache after this time, regardless of sliding expiration  
        .SetAbsoluteExpiration(TimeSpan.FromSeconds(5));  
}  
void createMemoryCache(object key, TResult cacheEntry, object paramKey, TParameter param)  
{  
    var cacheEntryOptions = createMemoryCacheEntryOptions();  

    // Save data in cache.  
    _cache.Set(key, cacheEntry, cacheEntryOptions);  

    // Save param in cache.  
    _cache.Set(paramKey, param, cacheEntryOptions);  
}  

}
``

Here is your cache memory declaration:
static MemoryCacheWithPolicy<RequestQuery, string> _requestQueryCache = new MemoryCacheWithPolicy<RequestQuery, string>();
where [RequestQuery] type parameter is what will force to have another creation.

Here is the final use of it:
[HttpGet]
public async Task GetPageFromUriOrBody(RequestQuery requestQuery)
{
//this is a static nLog I used w/in the memory cache and in controller
//log(nameof(GetPageFromUriOrBody), nameof(requestQuery));
var responseResult = await _requestQueryCache.GetOrCreate(
nameof(GetPageFromUriOrBody)
, requestQuery
, (x) => getPageContent(x).Result);
return Request.CreateResponse(System.Net.HttpStatusCode.Accepted, responseResult);
}

?
Anonymous
0 points
5 years ago

Forgot to mention:
You have to implement your own equals override for your param object (!cacheParam.Equals(param))
;)

?
Anonymous
0 points
5 years ago

One more "tip" for [getPageContent] deledated method:
This is the signature:
async Task getPageContent(RequestQuery requestQuery)

?
Anonymous
0 points
6 years ago

Great article, thanks.

?
Anonymous
0 points
6 years ago

You fixed concurrent issue using dictionary of SemaphoreSlims. You can optimize it a bit: 1. use single static semaphore slim or some fixed size of semaphore slim and key.GetHashCode() % somaphoresCount (need to measure of course); 2. keep in cache tasks not values itself, in that case you can just put task into cache without await under lock; in that case all consumers can await on this task in parallel. As an example https://github.com/MaximTka...

?
Anonymous
0 points
6 years ago

Awesome Maxim, thanks for this reference! There's a single semaphore, so from first glance I don't fully understand how threads that ask for a different key won't wait. I'll look at it more later, Thanks!