One of the most commonly used patterns in software development is Caching. It’s a simple, but a very effective concept. The idea is to reuse operation results. When performing a heavy operation, we will save the result in our cache container. The next time that we need that result, we will pull it from the cache container, instead of performing the heavy operation again.
For example, to get a person’s Avatar you might need a trip to the database. Instead of performing that trip every time, we will save that Avatar in the cache, pulling it from memory every time you need it.
Caching works great for data that changes infrequently. Or even better, never changes. Data that constantly changes, like the current machine’s time shouldn’t be cached or you will get wrong results.
In-process Cache, Persistant in-process Cache, and Distributed Cache
There are 3 types of caches:
- In-Memory Cache is used for when you want to implement cache in a single process. When the process dies, the cache dies with it. If you’re running the same process on several servers, you will have a separate cache for each server.
- Persistent in-process Cache is when you back up your cache outside of process memory. It might be in a file, or in a database. This is more difficult, but if your process is restarted, the cache is not lost. Best used when getting the cached item is expensive, and your process tends to restart a lot.
- Distributed Cache is when you want to have shared cache for several machines. Usually, it will be several servers. With a distributed cache, it is stored in an external service. This means if one server saved a cache item, other servers can use it as well. Services like Redis are great for this.
We’re going to talk just about in-process cache.
Naive Implementation
Let’s create a very simple cache implementation in C#:
public class NaiveCache<titem>
{
Dictionary<object titem=""> _cache = new Dictionary<object titem="">();
public TItem GetOrCreate(object key, Func<titem> createItem)
{
if (!_cache.ContainsKey(key))
{
_cache[key] = createItem();
}
return _cache[key];
}
}</titem></object></object></titem>
Usage:
var _avatarCache = new NaiveCache<byte>();
// ...
var myAvatar = _avatarCache.GetOrCreate(userId, () => _database.GetAvatar(userId));</byte>
This simple code solves a crucial problem. To get a user’s avatar, only the first request will actually perform a trip to the database. The avatar data (byte[]
) is then saved in process memory. All following requests for the avatar will be pulled from memory, saving time and resources.
But, as most things in programming, nothing is so simple. The above solution is not good for a number of reasons. For one thing, this implementation is not thread-safe. Exceptions can occur when used from multiple threads. Besides that, cached items will stay in memory forever, which is actually very bad.
Here’s why we should be removing items from Cache:
- Cache can take up a lot of memory, eventually leading to an out-of-memory exceptions and crashes.
- High memory consumption can lead to GC Pressure (aka Memory Pressure). In this state, the garbage collector works more than it should, hurting performance.
- Cache might need to be refreshed if the data changes. Our caching infrastructure should support that ability.
To handle these problems, cache frameworks have Eviction policies (aka Removal policies). These are rules to have items removed from cache according to some logic. Common eviction policies are:
- Absolute Expiration policy will remove an item from cache after a fixed amount of time, no matter what.
- Sliding Expiration policy will remove an item from cache if it wasn’t accessed in a fixed amound of time. So if I set the expiration to 1 minute, the item will keep staying in cache as long as I use it every 30 seconds. Once I don’t use it for longer than a minute, the item is evicted.
- Size Limit policy will limit the cache memory size.
Now that we know what we need, let’s continue on to better solutions.
Better Solutions
To my great dismay as a blogger, Microsoft already created a wonderful cache implementation. This deprived me the pleasure of creating a similar implementation myself, but at least I have less work writing this blog post.
I’ll show you Microsoft’s solution, how to effectively use it, and then how to improve it in some scenarios.
System.Runtime.Caching/MemoryCache vs Microsoft.Extensions.Caching.Memory
Microsoft has 2 solutions 2 different NuGet packages for caching. Both are great. As per Microsoft’s recommendation
, prefer using Microsoft.Extensions.Caching.Memory
because it integrates better with Asp. NET Core. It can be easily injected
into Asp .NET Core’s dependency injection mechanism.
Here’s a basic example with Microsoft.Extensions.Caching.Memory
:
public class SimpleMemoryCache<titem>
{
private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions());
public TItem GetOrCreate(object key, Func<titem> createItem)
{
TItem cacheEntry;
if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
{
// Key not in cache, so get data.
cacheEntry = createItem();
// Save data in cache.
_cache.Set(key, cacheEntry);
}
return cacheEntry;
}
}</titem></titem>
Usage:
var _avatarCache = new SimpleMemoryCache<byte>();
// ...
var myAvatar = _avatarCache.GetOrCreate(userId, () => _database.GetAvatar(userId));</byte>
This is very similar to my own NaiveCache
, so what changed? Well, for one thing, this is a thread-safe implementation. You can safely call this from multiple threads at once.
The second thing thing is the MemoryCache
allows for all the eviction policies we talked about before. Here’s an example:
IMemoryCache with eviction policies:
public class MemoryCacheWithPolicy<titem>
{
private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions()
{
SizeLimit = 1024
});
public TItem GetOrCreate(object key, Func<titem> createItem)
{
TItem cacheEntry;
if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
{
// Key not in cache, so get data.
cacheEntry = createItem();
var cacheEntryOptions = new MemoryCacheEntryOptions()
.SetSize(1)//Size amount
//Priority on removing when reaching size limit (memory pressure)
.SetPriority(CacheItemPriority.High)
// Keep in cache for this time, reset time if accessed.
.SetSlidingExpiration(TimeSpan.FromSeconds(2))
// Remove from cache after this time, regardless of sliding expiration
.SetAbsoluteExpiration(TimeSpan.FromSeconds(10));
// Save data in cache.
_cache.Set(key, cacheEntry, cacheEntryOptions);
}
return cacheEntry;
}
}</titem></titem>
Let’s analyze the new additions:
SizeLimit
was added inMemoryCacheOptions
. This adds a size-based policy to our cache container. Size doesn’t have a unit. Instead, we need to set the size amount on each cache entry. In this case, we set the amount to 1 each time withSetSize(1)
. This means that the cache is limited to 1024 items.- When we reach the size limit, which cache item should be removed? You can actually set priority with
.SetPriority(CacheItemPriority.High)
. The levels are Low, Normal, High, and NeverRemove. SetSlidingExpiration(TimeSpan.FromSeconds(2))
was added, which sets sliding expiration to 2 seconds. That means if an item was not accessed in over 2 seconds it will be removed.SetAbsoluteExpiration(TimeSpan.FromSeconds(10))
was added, which sets absolute expiration to 10 seconds. This means than the item will be evicted within 10 seconds if it wasn’t already.
In addition to the options in the example, you can also set a RegisterPostEvictionCallback
delegate, which will be called when an item is evicted.
That’s a pretty comprehensive feature set. It makes you wonder if there’s even anything else to add. There are actually a couple of things.
Problems and Missing features
There are a couple of important missing pieces in this implementation.
- While you can set the size-limit, the caching doesn’t actually monitor gc pressure. If we did monitor it, we could tighten policies when the pressure is high, and loosen up policies when the pressure is low.
- When requesting the same item with multiple threads at the same time, the requests don’t wait for the first one to finish. The item will be created multiple times. For example, let’s say we are caching the Avatar, and getting an avatar from the database takes 10 seconds. If we request an avatar 2 seconds after the first request, it will check if the avatar is cached (it isn’t yet), and start another trip to the database.
As for the first problem of gc pressure: It’s possible to monitor GC pressure with several techniques and heuristics. This blog post is not about that, but you can read my article Find, Fix, and Avoid Memory Leaks in C# .NET: 8 Best Practices to learn of some helpful methods.
The second problem is easier to solve. In fact, here’s an implementation of MemoryCache
that solves it entirely:
public class WaitToFinishMemoryCache<titem>
{
private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions());
private ConcurrentDictionary<object semaphoreslim=""> _locks = new ConcurrentDictionary<object semaphoreslim="">();
public async Task<titem> GetOrCreate(object key, Func<task>> createItem)
{
TItem cacheEntry;
if (!_cache.TryGetValue(key, out cacheEntry))// Look for cache key.
{
SemaphoreSlim mylock = _locks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
await mylock.WaitAsync();
try
{
if (!_cache.TryGetValue(key, out cacheEntry))
{
// Key not in cache, so get data.
cacheEntry = await createItem();
_cache.Set(key, cacheEntry);
}
}
finally
{
mylock.Release();
}
}
return cacheEntry;
}
}</task></titem></object></object></titem>
Usage:
var _avatarCache = new WaitToFinishMemoryCache<byte>();
// ...
var myAvatar =
await _avatarCache.GetOrCreate(userId, async () => await _database.GetAvatar(userId));</byte>
With this, when trying to get an item, if the same item is in the middle of being created by another thread, you will wait for the other to finish first. Then, you will get the already cached item created by the other thread.
Explanation of the code
This implementation locks the creation of an item. The lock is specific to the key. For example, if we’re waiting to get Alex’s Avatar, we can still get cached values of John or Sarah on another thread.
The dictionary _locks
stores all the locks. Regular locks don’t work with async/await
, so we need to use SemaphoreSlim
.
There are 2 checks to see if the value is already cached if (!_cache.TryGetValue(key, out cacheEntry)). The one inside the lock is the one that ensures there’s a single creation. The one outside of the lock is for optimization.
When to use WaitToFinishMemoryCache
This implementation obviously has some overhead. Let’s consider when it’s even necessary.
Use WaitToFinishMemoryCache when:
- When the creation time of an item has some sort of cost, and you want to minimize creations as much as possible.
- When the creation time of an item is very long.
- When the creation of an item has to be ensured to be done once per key.
Don’t use WaitToFinishMemoryCache when:
- There’s no danger of multiple threads accessing the same cache item.
- You don’t mind creating the item more than once. For example, if one extra trip to the database won’t change much.
Summary
Caching is a very powerful pattern. It’s also dangerous and has its own complexities. Cache too much and you can cause GC pressure. Cache too little and you can cause performance issues. Then, there’s distributed caching, which is a whole new world to explore. That’s software development for you, always something new to learn.
I hope you enjoyed this post. If you’re interested in memory management, my next article is going to be about the dangers of GC pressure and techniques to prevent it, so keep following . Happy coding.
Anyone who is interested in boosting ASP.NET application performance can download this whitepaper to get to know about 4 ways to optimize performance and remove bottlenecks.
http://www.alachisoft.com/r...
NCache is a great product!
In your WaitToFinishMemoryCache implemetation you are using below code,
SemaphoreSlim mylock = _locks.GetOrAdd(key, k => new SemaphoreSlim(1, 1));
If ConcurrentDictionar.GetOrAdd simultaneously on different threads, valueFactory may be called multiple times, would not cause any issues ?
Hi,
The code ensures it will not be called multiple times on different threads for the same key. It can be called multiple times for different keys, in which case it depends on the delegate content.
Michael, GetOrAdd it not thread-safe, so madhu is right. You can make it thread-safe using Lazy. https://andrewlock.net/maki...
Thanks Luis, I read it now.
From my understanding, using Lazy ensures the delegate "new SemaphoreSlim(1, 1)" runs only once. But, even with current implementation it will always return the same SemaphoreSlim instance for all threads, which is what matters. So it does't matter if the lock creation delegate runs more than once. It does matter if the value creation delegate runs more than once, but this is not the case.
So you're right that GetOrAdd can run the delegate more than once, but it's OK for our purposes.
Hi Michael, I agree with you. In this case it's not worth to use lazy. I've pointed out that GetOrAdd is not thread-safe because not all are aware of it and it can end with unexpected side effects.
Luis, you've actually solved an issue for me as to why the Microsoft MemoryCache is not thread-safe even though it uses ConcurrentDictionary internally, so Thanks!
Just an observation.... In the past 10 years I have been brought in to address "performance issues" at many clients. In three recent occasions, the client teams had worked hard to "cache and optimize" object lifetimes. There code looked well thought out, and they could even produce graphs that showed they helped performance....
My firm ripped all of that out, and minimized the implementations to "create, use, let-go" with the focus on the shortest object lifecycle possible. Yes, these even meant removing caching of semi-fixed lookup data from the database! The results was that overall application performance improved!
In summary, Michael' comment of "Caching is a very powerful pattern. It's also dangerous and has its own complexities." and the results can be quite counter intuitive.
Cool story David. That just goes to show how over optimization or incorrect optimization can actually make things worst.
For async operation better use https://docs.microsoft.com/... which now include memory realization.
Thanks for the tip Denis
Great article! One more thing is missing from your implementation: locks never removed from the _locks ConcurrentDictionary, so it can grow unlimited and can cause out of memory exception.
Thanks and nice catch!
Was this updated to prevent unlimited growth.. didn't see the article before this so I'm not aware of changes.
Nope, this is standalone
That was the first thing that came to mind when I saw that extra ConcurrentDictionary in the code. Funny since the article explicitly emphasizes the need to remove objects from cache and not have them linger around forever :-)
Good post! I was looking for a memory cache implementation to attach it in a repository pattern. The truth is that seeing the code seems an elegant way to solve the problem of concurrence in memory access. I have seen in the N-Ware commentary the problem of not freeing the memory of the SemaphoreSlim dictionary _lock, and investigating a bit I found a CallBack (PostEvictionCallBack) launched from the MemoryCache itself when a record is deleted. In case it helps.
Again nice post, and very usefull! :)
I agree that callback can be a good place for cleanup. I'll update the article when I have some time.
And thanks!
Ideally, we should avoid creating our own GetOrCreate method when IMemoryCache already comes with an extension method called... GetOrCreate! Granted, there are some unresolved issues with that (https://github.com/aspnet/E...), although they are partially contradicted in a similar discussion (https://github.com/aspnet/C...). In any case, I expect the issues will be resolved soon. Hopefully they will make the Core 3.0 release.
Depending on how the issues are resolved, the resulting behavior might be like ConcurrentDictionary: a race condition may lead to multiple values being created, but only one value is ever stored and returned. If it is truly undesirable to create a value multiple times (and often it does not really matter, as a one-time cost!), we can simply wrap the cached value in a Lazy (with the right LazyThreadSafetyMode), and then only one value will ever be created.
Timo I agree and hope this will be resolved within the library. Thanks for the links
Why class WaitToFinishMemoryCache is not static?
When should the cache be used as a singleton or scoped DI instance and what are the factors to consider? If creating the cache as a singleton if its used with any data layer that is scoped that would not work but is there performance issues if a concurrent dictionary is scoped where it would recreate the cache in memory each time?
You can decouple your memory cache from your model and still have implemented an async method.
Here is your modified code:
public class MemoryCacheWithPolicy<TParameter, TResult>
{
static ILogger _nlogger = new AppLogger().Logger;
private MemoryCache _cache = new MemoryCache(new MemoryCacheOptions()
{
SizeLimit = 1024
});
`` /// <summary>
/// Gets or creates a new memory cache record for a main data
/// along with parameter data that is assocciated with main main.
/// </summary>
/// <param name="key">Main data cache memory key.</param>
/// <param name="param">Parameter model that assocciated to main model (request result).</param>
/// <param name="createCacheData">A delegate to create a new main data to cache.</param>
/// <returns></returns>
public async Task<TResult> GetOrCreate(object key, TParameter param, Func<TParameter, TResult> createCacheData)
{
// this key is used for param cache memory.
var paramKey = key + nameof(param);
}
``
Here is your cache memory declaration:
static MemoryCacheWithPolicy<RequestQuery, string> _requestQueryCache = new MemoryCacheWithPolicy<RequestQuery, string>();
where [RequestQuery] type parameter is what will force to have another creation.
Here is the final use of it:
[HttpGet]
public async Task GetPageFromUriOrBody(RequestQuery requestQuery)
{
//this is a static nLog I used w/in the memory cache and in controller
//log(nameof(GetPageFromUriOrBody), nameof(requestQuery));
var responseResult = await _requestQueryCache.GetOrCreate(
nameof(GetPageFromUriOrBody)
, requestQuery
, (x) => getPageContent(x).Result);
return Request.CreateResponse(System.Net.HttpStatusCode.Accepted, responseResult);
}
Forgot to mention:
You have to implement your own equals override for your param object (!cacheParam.Equals(param))
;)
One more "tip" for [getPageContent] deledated method:
This is the signature:
async Task getPageContent(RequestQuery requestQuery)
Great article, thanks.
You fixed concurrent issue using dictionary of SemaphoreSlims. You can optimize it a bit: 1. use single static semaphore slim or some fixed size of semaphore slim and key.GetHashCode() % somaphoresCount (need to measure of course); 2. keep in cache tasks not values itself, in that case you can just put task into cache without await under lock; in that case all consumers can await on this task in parallel. As an example https://github.com/MaximTka...
Awesome Maxim, thanks for this reference! There's a single semaphore, so from first glance I don't fully understand how threads that ask for a different key won't wait. I'll look at it more later, Thanks!