For a project I’m working I needed a http client. Erlang has a couple like gun or hackney. However, I’ve noticed projects like erlcloud and rebar3 use httpc. If there are other http client libraries why should I use httpc? After some searching, I found this post on Erlang Forums. It seems that httpc is worth considering. So I decided to give it a go. Turns out it was a little more complicated so I thought a blog post walking through how to use it would be helpful. I’ll start with a basic version and progressively get to a more production ready form of it.

Basic Version

We will start with this simple function:

request(Method, Uri) ->
    request(Method, Uri, []).

request(Method, Uri, Headers) ->
    do_request(Method, {Uri, Headers}).

request(Method, Uri, Headers, ContentType, Body) ->
   do_request(Method, {Uri, Headers, ContentType, Body}).

do_request(Method, Request) ->
    handle_response(httpc:request(
        Method, 
        Request,
        [{ssl, [{verify, verify_none}]}],
        []
    )).

handle_response(Resp) ->
    case Resp of
        {ok, {_, RespHeaders, RespBody}} ->
            {ok, RespHeaders, RespBody};
        {error, _Reason} = E -> E
    end.

This will work for basic stuff but it’s susceptible to MITM attacks because we don’t do any verification. This behaviour is what we want for making requests to http like to localhost.

A basic GET request might look like this:

request(get, "http://localhost:8080/todos").

Maybe the API requires an Authorization header:

request(get, "http://localhost:8080/todos", [{"Authorization", "Bearer secretkey"}]).

To make a POST request with a JSON body is slightly more complicated:

request(
    post, 
    "http://localhost:8080/todo"
    [{"Content-Type", "application/json"}],
    "application/json",
    <<"{\"body\":\"Buy Groceries\"}">>
).

Notice we had to set the Content-Type header and tell httpc the content type as well. A bit weird, but whatever we can make it work. You could change the function to check for the Content-Type header and just use that. This works fine though.

Secure Version

Now we need to make this function secure. This turns out to be a little more complicated than I expected. Shoutout to @feld@friedcheese.us for giving me some tips on Mastodon where I shared the first version of this post. You can check out the discussion here. This next section includes the recommended updates.

First we need to add a few dependencies to our rebar.config:

{deps, [certifi, ssl_verify_fun, inet64_tcp]}.

Don’t forget to add these as applications to your app.src file.

• inet64_tcp: ensures that IPV6 connections will work
• certifi: has utilities to get certificate file across operating systems
• ssl_verify_fun: is used to verify the hostname is covered by the certificate

Then we will make a helper function:

ssl_opts(Hostname) ->
    {ssl, [
        {verify, verify_peer},
        {cacertfile, certifi:cacertfile()},
        {server_name_indication, Hostname},
        {reuse_sessions, false},
        {verify_fun, {fun ssl_verify_hostname:verify_fun/3, []}},
        {depth, 99}]}.

These options will make the connection verify the response is coming from the requested host.

Let’s use this helper function in our do_request function:

do_request(Method, {Uri, _, _, _}) ->
    #{host := Hostname} = uri_string:parse(Uri),
    handle_response(httpc:request(
      Method,
      Request,
      [ssl_opts(Hostname)],
      Body
    )).

Now we can make requests across the web in a secure way.

request(get, "https://example.com").

However, we can’t make requests to our localhost. What if we need to both?

Final Version

Let’s modify the ssl_opts function to take the URI and check if it is http or https.

Luckily Erlang has a builtin url parser function that will help with this.

We will need another helper function to get the protocol scheme and hostname from the URL.

url_scheme_hostname(URL) ->
    case uri_string:parse(URL) of
        #(scheme := Scheme, host := Hostname) ->
            {scheme host}
        _ ->
            undefined
    end.

Let’s use this function as part of ssl_opts.

ssl_opts(URL) ->
    Options =
    case get_scheme_hostname(URL) of
        {"https", Hostname} ->
            [{verify, verify_peer},
             {cacertfile, certifi:cacertfile()},
             {server_name_indication, Hostname},
             {reuse_sessions, false},
             {verify_fun, {fun ssl_verify_hostname:verify_fun/3, []}},
             {depth, 99}]
        _ ->
            [{verify, verify_none}]
    end,
    {ssl, Options}.

Basically we use the verification options if it is https and if its anything else we don’t verify. Now we can use the same request function for https or http.

With all this you should be set to get started with httpc and make your own http requests.