Terraform AzureRM too many requests (429)

Solving 429 http error using terraform on Azure Management API

Advertisements

When adding more resources to my azurerm terraform project, I encountered some concurrency problems. Terraform sends to many concurrent requests to azure. The Azure API gives a 429 http error. The fast solution is just to rerun your terraform apply, but that will not work in an automated environment (Infrastructure as Code). In this blog post I’ll show the error and what you can do to avoid it.

Http 429 Error

When deploying multiple vm’s (6) with terraform the following error occurred:

Error applying plan:
2 error(s) occurred:
* azurerm_network_interface.test6: autorest:DoErrorUnlessStatusCode 429 PUT https://management.azure.com/subscriptions/9bdde4e2-946e-4961-b111-fadcea94577d/resourceGroups/MySecondApp/providers/Microsoft.Network/networkInterfaces/test6?api-version=2015-06-15 failed with 429
* azurerm_network_interface.test2: autorest:DoErrorUnlessStatusCode 429 PUT https://management.azure.com/subscriptions/9bdde4e2-946e-4961-b111-fadcea94577d/resourceGroups/MyFirstApp/providers/Microsoft.Network/networkInterfaces/test2?api-version=2015-06-15 failed with 429
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

The http error code is part of RFC 6585:

The 429 status code indicates that the user has sent too many
   requests in a given amount of time ("rate limiting").

So terraform is sending more http request to azure than azure accepts in the given amount of time. There is a terraform issue on the GitHub.

Running without errors

Two possible solutions can be:

  1. Limit the requests by terraform
  2. Let Azure API accept more requests

Limiting the request by terraform can be done bij setting the parallelism parameter in the apply command. The default parallelism value is 10. Depending on the templates I had to set it to 2 to get the rid of the error.  This really limits the performance of your deployments

terraform apply -parallelism=2

The real solution would be to get the Management API accept more requests. I’m in the process of getting the requests limits for a subscription and how to change this with Microsoft.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s