Tips for Using Curl as a REST Client

RESTful interfaces make interacting with modern applications easier for other applications. “Easy” means you can use a variety of languages and tools to build scripts and automation around the application and that REST API, instead of relying only on manual clicks and a GUI.

You may already be familiar with curl for downloading files from websites. But it is also a flexible and convenient REST client! This post will walk through the curl options and invocations I find most useful.

The big advantage of curl is that it’s often already available on your machine or easily installed on a wide variety of systems. This is especially valuable when using containers because it keeps images simple.

For container environments, curl can be run from a stock CentOS container like this:

> docker run -it --rm centos:8 curl --version

For another useful guide on using curl as a REST client, see here.

Basic Curl Downloads to File or STDOUT

The basic and most widely-used mode for curl is to simply download locally a file from a remote HTTP server.

The following example command downloads a file from a remote location:

curl -O http://somewebsite.com/path/payload.tar

Note that by default curl outputs to STDOUT, so the “-O” flag signals to write the output to a local file with a filename that matches the last component of the path.

It may seem weird that by default curl outputs to STDOUT, but this actually means it fits in perfectly with the rest of the linux toolkit and mantra: “do one thing and do it well.” For example, you can easily download and search for strings by piping to grep. Or uncompress before writing locally.

The following line searching for a specific word ‘carping’ amongst a text file of Shakespeare’s plays:

curl https://ocw.mit.edu/ans7870/6/6.006/s08/lecturenotes/files/t8.shakespeare.txt | grep carping

You could also download a compressed file and decompress before saving local like this:

curl https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/$PACKAGE.tgz | tar -xvz -C /opt

Curl connecting easily to the rest of the linux utility belt will prove useful later.

Finally, notice that the above basic operations to download a file from a website are actually simple GET calls. These examples of downloading from a webpage actually utilize simple GET API calls.

Rest Client: Useful Command Flags

The following curl command-line flags are important for use as a REST client.

-X — specify HTTP method

Use a different REST verb than the default GET. For example you would use this to specify a PUT, POST, and DELETE call.

For example, the following command deletes an Elasticsearch index called ‘myindex’ by using the DELETE verb.

curl -X DELETE localhost:9200/myindex

-h — add header

Adds custom HTTP headers to the HTTP request.

For example, a common header to add is “Content-Type” which specifies to the server the data format being sent:

-H 'Content-Type: application/json'

Another common usage of headers is to include an authorization token.

-d — Provide data content

For many applications, you may need to send data in a PUT/POST/PATCH call and you can provide that data to curl from either the command line, a file, or STDIN.

As an example, to provide JSON content directly on the command line:

-d '{"type": "s3"}'

In addition to specifying data on the command line, you can use either file contents or STDIN.

• Read data from a file with “@filename”
• Read data from stdin with “@-”

To send the sequence of numbers 1..100 via STDIN to an NSQ service, see the following example:

> seq 100 | curl -d @- -XPUT 'http://nsqd:4151/pub?topic=test'

-sS — Limit verbosity

Hides the progress bar for downloads and uploads (-s) but still report errors (-S). I often find this to be the level of verbosity I want in larger scripts.

-k — Skip SSL verification
This option instructs curl to not verify SSL certificates for HTTPS connections. This is useful for services that use a self-signed certificate, which is common in internal and test environments. Without this flag, Curl correctly defaults to failing if the certificate cannot be verified. You should probably not be using this flag for production or public-facing applications.

-u — Specify basic authentication
Allows you to pass a username and password for basic authentication.

-u elastic:$es_password

Be careful using this method of specifying credentials as it is possible for other users to see your command line arguments on a shared system.

Controlling Retries and Timeouts

Curl also gives you fine-grained control over handling timeouts and retries.

curl
 --connect-timeout 5 # Max time to take making a connection
 --max-time 10       # Max time permitted for the whole operation
 --retry 5           # Number of times to retry
 --retry-delay 0     # Wait this amount of time between retries
 --retry-max-time 40 # Total time making a connection before failure
 http://myservice.internal/api

A combination of these options, especially the “max-time” parameter can help ensure that a slow or hung curl operation does not stall an entire batch job indefinitely.

Parsing JSON Outputs

Many REST services return answers in JSON format and the jq utility is useful for increasing the readability of the response with pretty printing. The simplest use-case involves piping the output returned from curl to jq with a single ‘.’ argument.

$ curl -Ss -k https://10.225.112.74/api/api_version
{“versions”:[“1.0”,”1.1",”1.2",”1.3",”1.4",”1.5",”1.6",”1.7",”1.8",”1.8.1",”1.9",”1.10"]}
$ curl -Ss -k https://10.225.112.74/api/api_version | jq .
{
  “versions”: [
   “1.0”,
   “1.1”,
   “1.2”,
   “1.3”,
   “1.4”,
   “1.5”,
   “1.6”,
   “1.7”,
   “1.8”,
   “1.8.1”,
   “1.9”,
   “1.10”
  ]
}

Even more advanced usages of jq allow for selecting and filtering specific JSON fields.

Elasticsearch Example

The following real example combines multiple elements to create a REST call that creates a new Elasticsearch snapshot repository:

curl -u “elastic:$PASSWORD” -k -X PUT “$ESHOST/_snapshot/my_s3_repository?pretty” -H 'Content-Type: application/json' -d'
{
  “type”: “s3”,
  “settings”: {
    “bucket”: “elastic-snapshots”,
    “endpoint”: “10.62.64.200”,
    “protocol”: “http”,
    “max_restore_bytes_per_sec”: “1gb”,
    “max_snapshot_bytes_per_sec”: “200mb”
  }
}
'

Another example from Elasticsearch deletes an index:

curl -X DELETE -u "elastic:$PASSWORD" -k "$ESHOST/oops_index?pretty"

Curl as an S3 Client

Curl can be used to connect to REST services like AWS S3, though I would recommend using a tool like s3curl or this example to help with the calculations necessary for signing requests with keys. If you do find yourself using this frequently, a purpose-built S3 tool like s5cmd might be suitable.

Summary

While there are many tools for making REST APIs, Curl is widely available and easy to use in scripts. This makes it perfect for ad hoc debug, administration, and automated testing. With a few handy options, Curl becomes a convenient REST API client.