Getting memory dump from .Net core Linux container on AKS

When running .Net core on Linux containers it should be easy to get an memory dump. However it is challenging to find a end to end guide on how to do this. We, Oscar Obeso and me, give you the steps in this blog post from indenting your issue, get a dump and than analyze the dump to get the root cause.

Monitoring

We are running our .Net core containers (3.x) on an AKS Linux cluster (1.18.x). To monitor the containers we have monitoring in place. On a dashboard (powerbi and azure alerts) we monitor the trends in calls, memory and cpu. Keeping track of the health enables us to identify issues before they are crashing your containers. When looking at our memory profile we noticed a gradual grow between new deployments. This could indicate a memory issue:

Memory monitoring on max memory for service pod in AKS cluster

You have a number of option on how to approach the issue. Check all your source code in review, try to reproduce locally or get a dump from a container where you know the issue is happening. In this guide we choose to go for getting a dump. This has the most change of identifying the issue correctly in a limited amount of time.

Get a dump

By following these steps you can get a dump from a .Net core container:

• Log on to azure
• Log on to the AKS cluster
• Get the running pods for finding something to dump
• Log on the pod
• Install tools
• Make dump (gc dump)
• Download dump

Log on to azure

az logon
az account set --subscription <subscription_guid>

log on to AKS cluster

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

Get the running pods for finding something to dump

kubectl get pods --namespace <namespace>

Log on to the pod

kubectl exec -it <pod_name> -c <container_name> --namespace <namespace> /bin/sh

Install tools (gc dump)

# Install bash
apk add bash

# Update wget. the pod already has it but it has an outdated version, and needs to be updated to install dotnet sdk
apk add wget

# Download DotnetSDK Installer
wget -O sdk_install.sh https://dot.net/v1/dotnet-install.sh

# Add permissions to file
chmod 777 sdk_install.sh

# Install sdk. -c argument also takes a 'Current' but it points to dotnet 5
./sdk_install.sh -c 3.1

# Go to the dotnet folder. In order to use SDK the process needs to be run directly
cd /root/.dotnet

# install dotnet gcdump or install dotnet-dump to get a full dump
./dotnet tool install --global dotnet-gcdump 

# Go to the tools folder
cd tools

Create dump

# This will list all the current processes where a garbage collector dump can be obtained from
./dotnet-gcdump ps 

# This will create the garbage collector dump for the given PID, in this case 1
./dotnet-gcdump collect -p 1 

# Exit the container, we are ready here
exit

Download dump

# Download the dump file from your pod to you local machine
kubectl exec -n <namespace> <pod_name> -- cat /root/.dotnet/tools/dumpresult1.gcdump > dumpresult1.gcdump

Analyze the dump file

When you downloaded the dump file you are can open it in Visual Studio. Then sort on memory usage, the root cause probably goes to the top of your list:

Visual Studio opens gcdump file

Conclusion

Getting started to get a dump file is far more intimating than actual getting one. Having the step to do so make it actually very easy. If you know the root cause, just fix you code and wait till the next time something gets out of control!

Advertisement

Azure Key Vault recover keys, secrets and certificates

Last week we had an incident in which we had deleted the wrong secret from our Azure Key Vault. After some research we found that it could have been recovered if we had used the Soft-delete in Key Vault. However, we did not know about this option and could not recover the item.

Run maintenance jobs on Azure SQL

Many users on Azure SQL Server do not realize they have to do their own maintenance on Indexes. This index will slowly become fragmented and the performance will decrease over time. Azure SQL does not have a Job scheduler (agent) like on premise. In this post I’ll describe how to schedule a maintenance job from an ASP.NET Core application.

Best practices using Azure Resource Manager templates

This article focuses on best practices regarding the automated deployment of resources to Azure. We have implemented Continuous Deployment (CD) pipelines including the provisioning of Azure resources for many customers, and we would like to share our experience so you can benefit from it. These practices will help you create more reliable, testable, reusable, and maintainable templates.

#TechdaysNL 2017 Best Practices ARM Templates

On October 12 2017 I did session on Best Practices ARM Templates at Microsoft TechdaysNL. The presentation shows some best practices we learned while using ARM templates. Download the pdf:

Best_Practices_ARM_Templates_TechDays2017

When I have the recordings of the presentation, I’ll add them to this post.

Thanks to the 160 enthusiast who where there.

Test Azure deployments in your VSTS Release Pipeline

When deploying Azure Resources you want to know if all resources are deployed as expected. To check if the resources are correctly deployed you can open the portal and visually inspect the deployed recourses or you can also run some powershell to validate the resources. Why not automate these checks and add them to your deployment pipeline. To validate the resources, I extended the Pester Build Task to connect to Azure. A test that checks the deployment of a VM can look like:

Serial copies in ARM Templates

You can now make copies in serial mode. That means that the copies are created after each other instead of parallel. This can be a good idea when you update a live resource. The resources will go down and up after each other.

A sample of this:

        {
            "apiVersion": "2015-01-01",
            "name": "[concat('nestedDeployment',copyIndex())]",
            "type": "Microsoft.Resources/deployments",
            "copy": {
                "name": "myCopySet",
                "count": 4,
                "mode": "serial",
                "batchSize": 2
            },

The mode and batchSize are new. Mode can be serial or parallel. The batchSize configures how many objects are created at the same time in serial mode.

New ways to support conditions in ARM Templates

Condition did in ARM templates where not that easy to implement. In the new Azure RM APIs a property condition is added. This condition makes many scenarios a lot easier to implement.

Add condition property to a resource object, the resource object will only be deployed when the condition is met. The following sample code will show this:

    "resources": [
        {
            "condition": "[equals(parameters('newOrExisting'),'new')]",
            "type": "Microsoft.Storage/storageAccounts",
            "name": "[variables('storageAccountName')]",
            "apiVersion": "2017-06-01",
            "location": "[resourceGroup().location]",
            "sku": {
                "name": "[variables('storageAccountType')]"
            },
            "kind": "Storage",
            "properties": {}
        },

The full code sample can be found at Ryan Jones GitHub.

The equals function will return a Boolean that is used as input for the condition.

This new feature in ARM Templates will make templates more readable, faster to develop and less need to copy one version of a template to different files to implement a condition.

Versioning ARM Template deployments

When deploying resources on Azure with Azure Resource Manager you want to be in control of which resources are deployed and control their life span. To get the control you need to do deploy in a tested, standardized and reusable manner. This can be done by managing your resource creation as Infrastructure as Code.

Azure Functions imperative bindings

The standard input and output bindings in Azure Functions are written in a declarative pattern using the function.json. When defining input and output declarative, you do not have the option to change some of the bindings properties like the name or make multiple outputs from one input. An imperative binding can do this for you. In this blog post I’ll show how to use imperative blob bindings.