Getting memory dump from .Net core Linux container on AKS

When running .Net core on Linux containers it should be easy to get an memory dump. However it is challenging to find a end to end guide on how to do this. We, Oscar Obeso and me, give you the steps in this blog post from indenting your issue, get a dump and than analyze the dump to get the root cause.

Monitoring

We are running our .Net core containers (3.x) on an AKS Linux cluster (1.18.x). To monitor the containers we have monitoring in place. On a dashboard (powerbi and azure alerts) we monitor the trends in calls, memory and cpu. Keeping track of the health enables us to identify issues before they are crashing your containers. When looking at our memory profile we noticed a gradual grow between new deployments. This could indicate a memory issue:

Memory monitoring on max memory for service pod in AKS cluster

You have a number of option on how to approach the issue. Check all your source code in review, try to reproduce locally or get a dump from a container where you know the issue is happening. In this guide we choose to go for getting a dump. This has the most change of identifying the issue correctly in a limited amount of time.

Get a dump

By following these steps you can get a dump from a .Net core container:

• Log on to azure
• Log on to the AKS cluster
• Get the running pods for finding something to dump
• Log on the pod
• Install tools
• Make dump (gc dump)
• Download dump

Log on to azure

az logon
az account set --subscription <subscription_guid>

log on to AKS cluster

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

Get the running pods for finding something to dump

kubectl get pods --namespace <namespace>

Log on to the pod

kubectl exec -it <pod_name> -c <container_name> --namespace <namespace> /bin/sh

Install tools (gc dump)

# Install bash
apk add bash

# Update wget. the pod already has it but it has an outdated version, and needs to be updated to install dotnet sdk
apk add wget

# Download DotnetSDK Installer
wget -O sdk_install.sh https://dot.net/v1/dotnet-install.sh

# Add permissions to file
chmod 777 sdk_install.sh

# Install sdk. -c argument also takes a 'Current' but it points to dotnet 5
./sdk_install.sh -c 3.1

# Go to the dotnet folder. In order to use SDK the process needs to be run directly
cd /root/.dotnet

# install dotnet gcdump or install dotnet-dump to get a full dump
./dotnet tool install --global dotnet-gcdump 

# Go to the tools folder
cd tools

Create dump

# This will list all the current processes where a garbage collector dump can be obtained from
./dotnet-gcdump ps 

# This will create the garbage collector dump for the given PID, in this case 1
./dotnet-gcdump collect -p 1 

# Exit the container, we are ready here
exit

Download dump

# Download the dump file from your pod to you local machine
kubectl exec -n <namespace> <pod_name> -- cat /root/.dotnet/tools/dumpresult1.gcdump > dumpresult1.gcdump

Analyze the dump file

When you downloaded the dump file you are can open it in Visual Studio. Then sort on memory usage, the root cause probably goes to the top of your list:

Visual Studio opens gcdump file

Conclusion

Getting started to get a dump file is far more intimating than actual getting one. Having the step to do so make it actually very easy. If you know the root cause, just fix you code and wait till the next time something gets out of control!

Advertisement

ASP .NET Core docker container warmup and liveness probes

When running your ASP.NET core container/application you probably noticed that the first requests take longer on average. The cause of the longer request can be normal application loading and/or logic you have written that must initialize on your first call. It would be nice to warm up your container before a customer call is handled by your container. Kubernetes gives you the possibility to use a readiness probe to check and warm up your application.

Hosting services in .NET Core console application

When building services for data processing you do not always need a user interface. An IHost is very capable of hosting such an application in a console application as headless service. The IHost does give you a number of advantages like graceful shut down, dependency injection, logging, and configuration. When running long processing tasks in Docker containers, graceful shut down helps you to keep the state of your application consistent. This post explains how making use of the generic IHost in .NET Core for headless services.

HttpClient and HttpClientFactory in ASP.NET Core

With .NET Core 2.1 the HttpClientFactory is introduced. The HttpClientFactory is a factory class which helps with managing HttpClient instances. Managing your own HttpClient correctly was not so easy. The HttpClientFactory gives you a number of options for easy management of your HttpClient instances. In this post I’ll explain how to use the HttpClientFactory in your ASP.NET Core application.

Access XML SOAP services in .NET Core and client certificates (SSL)

Only a few years back Windows Communication Foundation (WCF) was the way to do communication on the Microsoft platform based on SOAP protocol. Now a days new services are mostly build on top of Representational State Transfer (REST) Services. Sometimes you have to access a ‘legacy’ SOAP services for .NET Core. .NET Core has limited WCF support. In this blog post I’ll explain how to consume SOAP services form .NET Core.

Run scheduled background tasks in ASP.NET Core

In the previous blog post called background tasks with ASP.NET Core using the IHostedService Peter described how to use the IHostedInterface for background tasks. In this post, we continue on this subject and add some pointers on how to perform scheduled background tasks.

In many software projects, there are repetitive tasks; some do just repeat every x seconds after the last instance is finished but you might also have to run a task on a schedule like every 10 minutes. When building repeating or scheduled tasks there are many options on how to approach the scheduling and this approach can be influenced by a number of technical choices.

Building the scheduling yourself is an option when you do not want to add extra dependencies to your project, have full control or just want an extra technical challenge. An out of the box solution you can a look at Hangfire, Quartz.net, or an external service that does an http call every x seconds to trigger the task (something like Pingdom).

Upgrade ASP.NET Core and Entity Framework Core 2.0 to 2.1

In the past weeks we have upgraded our ASP.NET Core 2.0 project to 2.1. The main reason for the upgrade is using the latest signalr capabilities and hosted services. The issues we had are related to the features we use. In our case the upgrade encountered some minor issues. In this blog post I’ll show what we had to change and give some tips on how to upgrade.

Let your build work for you, shorten the feedback loop

Maximize how you use your VSTS build and release pipeline with Inline Powershell tasks. In this blog series ‘Tips and Tricks for Inline Powershell’, I will show simple samples on how to get more out of your pipelines. This blog post: Let your build work for you

VSTS Rest API from Inline Powershell

Maximize how you use your VSTS build and release pipeline with Inline Powershell tasks. In this blog series ‘Tips and Tricks for Inline Powershell’, I will show simple samples on how to get more out of your pipelines. This blog post: VSTS Rest API

Navigate VSTS project as filesystem from Inline Powershell

Maximize how you use your VSTS build and release pipeline with Inline Powershell tasks. In this blog series ‘Tips and Tricks for Inline Powershell’, I will show simple samples on how to get more out of your pipelines. This blog post: Navigate VSTS as filesystem