Getting memory dump from .Net core Linux container on AKS

When running .Net core on Linux containers it should be easy to get an memory dump. However it is challenging to find a end to end guide on how to do this. We, Oscar Obeso and me, give you the steps in this blog post from indenting your issue, get a dump and than analyze the dump to get the root cause.

Monitoring

We are running our .Net core containers (3.x) on an AKS Linux cluster (1.18.x). To monitor the containers we have monitoring in place. On a dashboard (powerbi and azure alerts) we monitor the trends in calls, memory and cpu. Keeping track of the health enables us to identify issues before they are crashing your containers. When looking at our memory profile we noticed a gradual grow between new deployments. This could indicate a memory issue:

Memory monitoring on max memory for service pod in AKS cluster

You have a number of option on how to approach the issue. Check all your source code in review, try to reproduce locally or get a dump from a container where you know the issue is happening. In this guide we choose to go for getting a dump. This has the most change of identifying the issue correctly in a limited amount of time.

Get a dump

By following these steps you can get a dump from a .Net core container:

• Log on to azure
• Log on to the AKS cluster
• Get the running pods for finding something to dump
• Log on the pod
• Install tools
• Make dump (gc dump)
• Download dump

Log on to azure

az logon
az account set --subscription <subscription_guid>

log on to AKS cluster

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

Get the running pods for finding something to dump

kubectl get pods --namespace <namespace>

Log on to the pod

kubectl exec -it <pod_name> -c <container_name> --namespace <namespace> /bin/sh

Install tools (gc dump)

# Install bash
apk add bash

# Update wget. the pod already has it but it has an outdated version, and needs to be updated to install dotnet sdk
apk add wget

# Download DotnetSDK Installer
wget -O sdk_install.sh https://dot.net/v1/dotnet-install.sh

# Add permissions to file
chmod 777 sdk_install.sh

# Install sdk. -c argument also takes a 'Current' but it points to dotnet 5
./sdk_install.sh -c 3.1

# Go to the dotnet folder. In order to use SDK the process needs to be run directly
cd /root/.dotnet

# install dotnet gcdump or install dotnet-dump to get a full dump
./dotnet tool install --global dotnet-gcdump 

# Go to the tools folder
cd tools

Create dump

# This will list all the current processes where a garbage collector dump can be obtained from
./dotnet-gcdump ps 

# This will create the garbage collector dump for the given PID, in this case 1
./dotnet-gcdump collect -p 1 

# Exit the container, we are ready here
exit

Download dump

# Download the dump file from your pod to you local machine
kubectl exec -n <namespace> <pod_name> -- cat /root/.dotnet/tools/dumpresult1.gcdump > dumpresult1.gcdump

Analyze the dump file

When you downloaded the dump file you are can open it in Visual Studio. Then sort on memory usage, the root cause probably goes to the top of your list:

Visual Studio opens gcdump file

Conclusion

Getting started to get a dump file is far more intimating than actual getting one. Having the step to do so make it actually very easy. If you know the root cause, just fix you code and wait till the next time something gets out of control!

Advertisement

ASP .NET Core docker container warmup and liveness probes

When running your ASP.NET core container/application you probably noticed that the first requests take longer on average. The cause of the longer request can be normal application loading and/or logic you have written that must initialize on your first call. It would be nice to warm up your container before a customer call is handled by your container. Kubernetes gives you the possibility to use a readiness probe to check and warm up your application.

ASP.NET Core Web API custom formatters

ASP.NET Core does support out of the box JSON, XML, or plain text formatters based on the ACCEPT Header. In this post, I’ll explain how to specify other formatters and return them based on the ACCEPT Header. As an example, I use a CSV formatter to return a CSV formatter. A controller method returns the format based on the ACCEPT Header which is specified by the client. This can be very useful in cases where you need to export your data to Excel, and the use of more formats makes your application more accessible.

HttpClient and HttpClientFactory in ASP.NET Core

With .NET Core 2.1 the HttpClientFactory is introduced. The HttpClientFactory is a factory class which helps with managing HttpClient instances. Managing your own HttpClient correctly was not so easy. The HttpClientFactory gives you a number of options for easy management of your HttpClient instances. In this post I’ll explain how to use the HttpClientFactory in your ASP.NET Core application.

.NET Core, what is my OS

There are a few cases where it is very useful to know the OS your .NET Core application is running on. One of those cases is when you need to set a time zone. Each OS does have its own naming of time zones. You can just try and catch the exception and then retry for another OS, however, if you know the OS, then it is far nicer to do this without exception handling.

Injecting a Scoped service into IHostedService

In our projects we are using IHostedService to run background processes. What I have seen is that when injecting a scoped service into the IHostedService you get a InvalidOperationException on startup. The first time you get this exception, it is takes some time to figure out what is going on. This blog post will help you to resolve the problem faster.

Run maintenance jobs on Azure SQL

Many users on Azure SQL Server do not realize they have to do their own maintenance on Indexes. This index will slowly become fragmented and the performance will decrease over time. Azure SQL does not have a Job scheduler (agent) like on premise. In this post I’ll describe how to schedule a maintenance job from an ASP.NET Core application.

Access XML SOAP services in .NET Core and client certificates (SSL)

Only a few years back Windows Communication Foundation (WCF) was the way to do communication on the Microsoft platform based on SOAP protocol. Now a days new services are mostly build on top of Representational State Transfer (REST) Services. Sometimes you have to access a ‘legacy’ SOAP services for .NET Core. .NET Core has limited WCF support. In this blog post I’ll explain how to consume SOAP services form .NET Core.

Run scheduled background tasks in ASP.NET Core

In the previous blog post called background tasks with ASP.NET Core using the IHostedService Peter described how to use the IHostedInterface for background tasks. In this post, we continue on this subject and add some pointers on how to perform scheduled background tasks.

In many software projects, there are repetitive tasks; some do just repeat every x seconds after the last instance is finished but you might also have to run a task on a schedule like every 10 minutes. When building repeating or scheduled tasks there are many options on how to approach the scheduling and this approach can be influenced by a number of technical choices.

Building the scheduling yourself is an option when you do not want to add extra dependencies to your project, have full control or just want an extra technical challenge. An out of the box solution you can a look at Hangfire, Quartz.net, or an external service that does an http call every x seconds to trigger the task (something like Pingdom).

Upgrade ASP.NET Core and Entity Framework Core 2.0 to 2.1

In the past weeks we have upgraded our ASP.NET Core 2.0 project to 2.1. The main reason for the upgrade is using the latest signalr capabilities and hosted services. The issues we had are related to the features we use. In our case the upgrade encountered some minor issues. In this blog post I’ll show what we had to change and give some tips on how to upgrade.