When adding more resources to my azurerm terraform project, I encountered some concurrency problems. Terraform sends to many concurrent requests to azure. The Azure API gives a 429 http error. The fast solution is just to rerun your terraform apply, but that will not work in an automated environment (Infrastructure as Code). In this blog post I’ll show the error and what you can do to avoid it.
Http 429 Error
When deploying multiple vm’s (6) with terraform the following error occurred:
Error applying plan: 2 error(s) occurred: * azurerm_network_interface.test6: autorest:DoErrorUnlessStatusCode 429 PUT https://management.azure.com/subscriptions/9bdde4e2-946e-4961-b111-fadcea94577d/resourceGroups/MySecondApp/providers/Microsoft.Network/networkInterfaces/test6?api-version=2015-06-15 failed with 429 * azurerm_network_interface.test2: autorest:DoErrorUnlessStatusCode 429 PUT https://management.azure.com/subscriptions/9bdde4e2-946e-4961-b111-fadcea94577d/resourceGroups/MyFirstApp/providers/Microsoft.Network/networkInterfaces/test2?api-version=2015-06-15 failed with 429 Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.
The http error code is part of RFC 6585:
The 429 status code indicates that the user has sent too many
requests in a given amount of time ("rate limiting").
So terraform is sending more http request to azure than azure accepts in the given amount of time. There is a terraform issue on the GitHub.
Running without errors
Two possible solutions can be:
- Limit the requests by terraform
- Let Azure API accept more requests
Limiting the request by terraform can be done bij setting the parallelism parameter in the apply command. The default parallelism value is 10. Depending on the templates I had to set it to 2 to get the rid of the error. This really limits the performance of your deployments
terraform apply -parallelism=2
The real solution would be to get the Management API accept more requests. I’m in the process of getting the requests limits for a subscription and how to change this with Microsoft.
{ Think Rethink } 