How to add User-Agent header to HttpClient in .NET

An intrinsic part of the modern web application is the ability to use external APIs and webpages. In .NET this is done using the HttpClient class. The HttpClient class does not add the field User-Agent out of the box. According to the HTTP standard, it is recommended that this is added to HTTP Requests. This is especially important when doing automated requests like a bot or a web crawler/scraper. In this article, I will show the format for the User-Agent header and how to add it individually to each request or by default.

The purpose

The purpose of the User-Agent header is to tell the API/website that you access what you are using to access the site, why you are accessing their site, who you are, and how they can get in contact with you. This can be useful if the owner of the site doesn't want you to access the specific site or if they have suggestions for things that you could do differently when requesting their resources. Some public API's also limit your bandwidth and requests/minute if you do not supply a valid User-Agent header.

The User-Agent format

The general format for a User-Agent value is as follows:

(<Product>/<Product version> (<Comments>))*

This means that it contains any number of product-version-comment pairs specifiying the products used to make the request one after each other with increasing specificity. The comments are optional and should be encased in parenthesis. In browsers, this is used to specify things like browser and render engine. When we make a request using HttpClient, we can specify what framework we use to make the request (this is .NET), and we should indicate that this is an automated request. An example of one for a webscraper could be as the following:

.NET/5.0 ScraperBot/1.0 (+http://www.example.com/ScraperBot.html)

We have specified that we use .NET 5.0, that we have made a bot called ScraperBot which we used for this request and as a comment a link to proceeded by a +-sign to a page where one can read more about the bot. That we use .NET 5.0 is optional to inform. It is common that "bot" is a part of the name to indicate that this is an automated request. The site that is linked to should be very simple and contain the name of the bot, the purpose, how they can contact the owner/creator, and when the read-more site has last been updated.

Adding User-Agent header to single HttpRequest

If you wish to add the User-Agent header to a single HttpRequestMessage, this can be done like this:

var httpClient = new HttpClient();

var request = new HttpRequestMessage(HttpMethod.Get, "https://API.com/api");

var productValue = new ProductInfoHeaderValue("ScraperBot", "1.0");
var commentValue = new ProductInfoHeaderValue("(+http://www.API.com/ScraperBot.html)");

request.Headers.UserAgent.Add(productValue);
request.Headers.UserAgent.Add(commentValue);

var resp = await httpClient.SendAsync(request);

We create a new request. Then we add two fields for the User-Agent header. One which defines the product and one that defines the comment. The format for the string of the comment is only validated on runtime, so remember to add the parenthesis around the comment.

Adding it as default to HttpClient.

Sometimes you need the same header for many requests during the instance of a single HttpClient. For this, we can add the User-Agent header as a default header to the HttpClient.

var httpClient = new HttpClient();

var productValue = new ProductInfoHeaderValue("ScraperBot", "1.0");
var commentValue = new ProductInfoHeaderValue("(+http://www.example.com/ScraperBot.html)");

httpClient.DefaultRequestHeaders.UserAgent.Add(productValue);
httpClient.DefaultRequestHeaders.UserAgent.Add(commentValue);

var request = new HttpRequestMessage(HttpMethod.Get, "https://API.com/api");
var resp = await httpClient.SendAsync(request);

Adding default headers to HttpClientFactory

A common way to create HttpClients in WebAPI and MVC projects for .NET is using a HttpClientFactory. This can be added to the Service Collection in Startup.cs and be Dependency Injected in each controller.
First, we add the HttpClientFactory in Startup.cs.

public void ConfigureServices(IServiceCollection services)
{
    // Other services

    services.AddHttpClient("Bot", config =>
    {
        var productValue = new ProductInfoHeaderValue("ScraperBot", "1.0");
        var commentValue = new ProductInfoHeaderValue("(+http://www.example.com/ScraperBot.html)");

        config.DefaultRequestHeaders.UserAgent.Add(productValue);
        config.DefaultRequestHeaders.UserAgent.Add(commentValue);
    });
}

We have added the HttpClientFactory with a specific configuration for the name "Bot". This can be used later to create a HttpClient with these settings.
Now, we can inject the HttpClientFactory in a controller.

public class ExampleController : ControllerBase
{
    private readonly IHttpClientFactory httpClientFactory;

    public ExampleController(IHttpClientFactory httpClientFactory)
    {
        this.httpClientFactory = httpClientFactory;
    }

    public async Task<ActionResult> Example()
    {
        var httpClient = httpClientFactory.CreateClient("Bot");

        var request = new HttpRequestMessage(HttpMethod.Get, "https://example.com/api");
        var resp = await httpClient.SendAsync(request);
    }
}

In the action that we want to use the HttpClient, we call the CreateClient method with the name "Bot" as argument to get our pre-configured HttpClient.

Conclusion

I have explained what the purpose is for the User-Agent header. I have shown, the general format of the User-Agent header. And in the end, I have shown how to add this header to HttpRequests in .NET. I have shown how to do this in 3 different ways: on the HttpRequestMessage, on the HttpClient, and via the HttpClientFactory dependency injector. If you have any questions regarding this article or feedback then please reach out.

See how we can help you monitor your website for crashes Monitor your website