Getting memory dump from .Net core Linux container on AKS

When running .Net core on Linux containers it should be easy to get an memory dump. However it is challenging to find a end to end guide on how to do this. We, Oscar Obeso and me, give you the steps in this blog post from indenting your issue, get a dump and than analyze the dump to get the root cause.

Monitoring

We are running our .Net core containers (3.x) on an AKS Linux cluster (1.18.x). To monitor the containers we have monitoring in place. On a dashboard (powerbi and azure alerts) we monitor the trends in calls, memory and cpu. Keeping track of the health enables us to identify issues before they are crashing your containers. When looking at our memory profile we noticed a gradual grow between new deployments. This could indicate a memory issue:

Memory monitoring on max memory for service pod in AKS cluster

You have a number of option on how to approach the issue. Check all your source code in review, try to reproduce locally or get a dump from a container where you know the issue is happening. In this guide we choose to go for getting a dump. This has the most change of identifying the issue correctly in a limited amount of time.

Get a dump

By following these steps you can get a dump from a .Net core container:

• Log on to azure
• Log on to the AKS cluster
• Get the running pods for finding something to dump
• Log on the pod
• Install tools
• Make dump (gc dump)
• Download dump

Log on to azure

az logon
az account set --subscription <subscription_guid>

log on to AKS cluster

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

Get the running pods for finding something to dump

kubectl get pods --namespace <namespace>

Log on to the pod

kubectl exec -it <pod_name> -c <container_name> --namespace <namespace> /bin/sh

Install tools (gc dump)

# Install bash
apk add bash

# Update wget. the pod already has it but it has an outdated version, and needs to be updated to install dotnet sdk
apk add wget

# Download DotnetSDK Installer
wget -O sdk_install.sh https://dot.net/v1/dotnet-install.sh

# Add permissions to file
chmod 777 sdk_install.sh

# Install sdk. -c argument also takes a 'Current' but it points to dotnet 5
./sdk_install.sh -c 3.1

# Go to the dotnet folder. In order to use SDK the process needs to be run directly
cd /root/.dotnet

# install dotnet gcdump or install dotnet-dump to get a full dump
./dotnet tool install --global dotnet-gcdump 

# Go to the tools folder
cd tools

Create dump

# This will list all the current processes where a garbage collector dump can be obtained from
./dotnet-gcdump ps 

# This will create the garbage collector dump for the given PID, in this case 1
./dotnet-gcdump collect -p 1 

# Exit the container, we are ready here
exit

Download dump

# Download the dump file from your pod to you local machine
kubectl exec -n <namespace> <pod_name> -- cat /root/.dotnet/tools/dumpresult1.gcdump > dumpresult1.gcdump

Analyze the dump file

When you downloaded the dump file you are can open it in Visual Studio. Then sort on memory usage, the root cause probably goes to the top of your list:

Visual Studio opens gcdump file

Conclusion

Getting started to get a dump file is far more intimating than actual getting one. Having the step to do so make it actually very easy. If you know the root cause, just fix you code and wait till the next time something gets out of control!

ASP .NET Core docker container warmup and liveness probes

When running your ASP.NET core container/application you probably noticed that the first requests take longer on average. The cause of the longer request can be normal application loading and/or logic you have written that must initialize on your first call. It would be nice to warm up your container before a customer call is handled by your container. Kubernetes gives you the possibility to use a readiness probe to check and warm up your application.

Announce changes with ItemStatus WPF

After announcing changes on the screen with LifeRegionChanged, I found how I can use ItemStatus to do something similar for the focused element. Using ItemStatus can be used to keep track of the status of an element and announce status changes on the element. In this post, I’ll give a sample of validation on a TextBox and explain how to use AccEvent.exe tool to see the accessibility events.

Narrator announce status changes

In this blog post, I want to discuss how to make status changes on the screen accessible to users with a screen reader. In this case, the user started a process and as time passed, the process when through a number of states. When the state changes the narrator should announce that to the user.

Narrator reads collapsed content

Last week, I worked on some accessibility issues. One of them was the windows narrator reading collapsed content. Hidding the content from the narrator was a little harder than I would have expected when starting working on the issues.

How many workers in a Docker cluster

Recently we have created a k8s cluster to host our Microservices architecture. In this blog post, I like to share how we calculated the number of worker machines we needed and which considerations are to take into account.

ASP.NET Core Web API custom formatters

ASP.NET Core does support out of the box JSON, XML, or plain text formatters based on the ACCEPT Header. In this post, I’ll explain how to specify other formatters and return them based on the ACCEPT Header. As an example, I use a CSV formatter to return a CSV formatter. A controller method returns the format based on the ACCEPT Header which is specified by the client. This can be very useful in cases where you need to export your data to Excel, and the use of more formats makes your application more accessible.

Hosting services in .NET Core console application

When building services for data processing you do not always need a user interface. An IHost is very capable of hosting such an application in a console application as headless service. The IHost does give you a number of advantages like graceful shut down, dependency injection, logging, and configuration. When running long processing tasks in Docker containers, graceful shut down helps you to keep the state of your application consistent. This post explains how making use of the generic IHost in .NET Core for headless services.

HttpClient and HttpClientFactory in ASP.NET Core

With .NET Core 2.1 the HttpClientFactory is introduced. The HttpClientFactory is a factory class which helps with managing HttpClient instances. Managing your own HttpClient correctly was not so easy. The HttpClientFactory gives you a number of options for easy management of your HttpClient instances. In this post I’ll explain how to use the HttpClientFactory in your ASP.NET Core application.

Azure Key Vault recover keys, secrets and certificates

Last week we had an incident in which we had deleted the wrong secret from our Azure Key Vault. After some research we found that it could have been recovered if we had used the Soft-delete in Key Vault. However, we did not know about this option and could not recover the item.