Developers using Azure Application Insights face the challenge of monitoring their apps’ performance while keeping costs down due to the vast amount of data generated. In this blog, I share how I reduced logging costs for my 100-500 MB per day side project that already costs around $20/mo, and is expected to grow larger.
Reducing/filtering dependency logs
The first thing I tried to improve was the big number of dependency logs. Most of these logs were calls to my database. For a side project, these logs are not that important to me. Let’s only log them when an error happens.
If you want to filter which telemetry makes it to your Application Insights or Log Analytics Workspace, you can implement a custom ITelemetryProcessor. To do this, first we need to install a NuGet package: Microsoft.ApplicationInsights.
Then, we create a new class that implements the ITelemetryProcessor. This interface exposes the Process method, in which we can do the filtering. In the following code example, you can see that I check if the ITelemtry item is of type DependencyTelemtry. If this is the case, I check whether the telemtry was successful. If it’s successful, I choose to ignore it by returning the void method. Otherwise, we let it go through.
This is just a really simple example. In this TelemtryProcessor we can also filter on different types of telemetry like RequestTelemetry, ExceptionTelemetry, etc.
public class CustomTelemetryProcessor : ITelemetryProcessor
{
public CustomTelemetryProcessor(ITelemetryProcessor next)
{
Next = next;
}
private ITelemetryProcessor Next { get; set; }
public void Process(ITelemetry item)
{
if (item is DependencyTelemetry dependency)
{
if (dependency.Success)
{
return;
}
}
Next.Process(item);
}
}
To start using this CustomTelemetryProcessor, add the following line(s) to your Program.cs or Startup.cs.
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddApplicationInsightsTelemetryProcessor<CustomTelemetryProcessor>();
Reducing logs to only exceptions
After reducing the number of dependency logs, I found that many unnecessary logs still remained. I decided to go all in, deleting all the logs I didn’t need. For a side project, this basically meant everything but the exceptions.
However, removing all the logs also meant that during debugging I didn’t see any more logs. I didn’t like this, so I started looking into a solution. Luckily, I found one. We’ll go over the solutions for a .NET 6 Web App and Function App.
.NET 6 Web App
Every app has a appsettings.json file. This is where we should also put our log settings. For my own project, I set the Default namespace to Error. The Default namespace is the namespace it falls back to when the specific namespace is not defined. If you want logging for some namespaces to stay on, please add them next to the default one. The code looks like this.
"Logging": {
"LogLevel": {
"Default": "Error",
"Some.Random.NameSpace": "Information"
}
}
However, adding the above code snippet to your appsettings.json will also block logging locally. How did I fix this? For my local development, I decided to add a new appsettings file: AppSettings.LocalDevelopment.json. In this file I save all settings that I want to override from my Azure App Configuration or Key Vault. Here I override my Default namespace logging to Information.
To add this config file to your IConfiguration, add the following code to your Program.cs.
var builder = WebApplication.CreateBuilder(args);
builder.Configuration
.AddAzureConfigAndKeyVault(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production")
#if DEBUG
.AddJsonFile("appsettings.LocalDevelopment.json", false, true)
#endif
.Build();
.NET 6 Functions App
Settings log levels in a function app works a bit different. Here we have to work with the host.json file. Where we can let log levels for single functions or the entire function project. I copied the appsettings.json code from above and pasted it in my host.json. However, same story here. It blocks logging locally. To fix this, simply add the following line of code to your local.settings.json. Or use the same approach by adding a appsettings.LocalDevelopment.json to your project.
"AzureFunctionsJobHost__logging__LogLevel__Default": "Information"
The AzureFunctionsJobHost part refers to the host.json and will override it for you.
I hope you learned something from this blog. If you want to find out where your biggest logging costs are coming from, please head over to my friend and fellow blogger Mart de Graaf’s blog!
Until the next! 🙂
