We’ve opened a new DevOps (Bash, LAMP and LEMP Stack) Professional position. Part-time and entirely remote, of course.
1. Overview
The cURL tool is a well-known tool for transferring data across networks, based on the libcurl library. It’s pre-installed on many Linux distributions. Furthermore, we can find curl in the repositories of most distributions. Besides the HTTP(S) protocol, many other network protocols are supported, like FTP, IMAP, LDAP, and SMB.
In this tutorial, we’ll learn how to use the curl command within the Linux operating system (OS).
2. Working With URLs
The curl command accepts one or more URLs. The command sends a GET HTTP request to the defined URL if we don’t set any options.
Let’s download an example web page:
$ curl http://www.example.com
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...Here, we requested the content of a web page, without providing any options. The content of the page is printed to the terminal.
Further, we can enter more than one URL. In this case, curl downloads the pages sequentially:
$ curl http://www.example.com/ http://www.example.edu
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...
Indeed, we can see that both responses are printed in the terminal, one after the other.
Also, curl attempts to reuse connections for URLs that refer to the same server. It performs this optimization only within a single command call. As a result, multiple downloads within a single command call may have a better performance than splitting them into many command calls.
If we don’t specify a protocol in the URL, curl tries to guess the protocol:
$ curl www.example.com
<!doctype html>
<html>
<head>
...As expected, curl had no trouble fetching the content, even without setting the protocol in the URL.
3. Adding Options
Options start with one or two dash characters. If an option isn’t a boolean, it’s followed by a space and a value. However, the short form of an option starts with a single dash and the space before the value is optional.
For example, let’s use the –output option, which saves the response in a file instead of printing it to the terminal:
$ curl --output response.html www.example.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 648 100 648 0 0 30596 0 --:--:-- --:--:-- --:--:-- 30857
$ cat response.html
<!doctype html>
<html>Indeed, the response is stored in the response.html file and the command printed some download information in the terminal instead.
The short form of the –output option is -o:
$ curl -oresponse.html www.example.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1256 100 1256 0 0 65549 0 --:--:-- --:--:-- --:--:-- 66105As expected, the outcome is the same.
On the other hand, we can disable any boolean option with the –no prefix.
For example, the –insecure option skips the server’s TLS certificate verification. We may disable this option using the –no prefix:
$ curl --no-insecure https://www.example.comOf course, the server certificate verification is enabled by default, so we’re not required to add this option.
4. Exit Codes
The curl command returns a zero exit code when it’s successful. If there’s an error, the command returns a non-zero exit code. Furthermore, there are about a hundred different exit codes that describe various errors.
For example, let’s use an unsupported protocol:
$ curl httpk://www.example.com
curl: (1) Protocol "httpk" not supported or disabled in libcurl
$ echo $?
1As expected, the exit code is 1, corresponding to the unsupported protocol error.
Another error is the malformed URL:
$ curl http://$#!
curl: (3) URL using bad/illegal format or missing URL
$ echo $?
3Here, the command returned a different exit code, 3 in particular. This exit code corresponds to a malformed URL error.
5. Examples
Let’s see some common or potentially useful examples of the curl command.
5.1. Setting a Proxy
We can use the –proxy option to set a proxy:
$ curl --proxy http://<myproxyhost>:<myproxyport> http://www.example.comAlternatively, the short form of the option is -x:
$ curl -x http://<myproxyhost>:<myproxyport> http://www.example.comAlso, if the proxy requires authentication, we can add the string username:password@ before the proxy host and after the protocol.
Using a proxy to access a given resource is useful, as we won’t need to use other means to ensure the given curl call goes through the required proxy.
5.2. Trace Request and Response
The –trace and –trace-ascii options store a trace of the request and the response in a specified file.
In practice, the –trace option gets the trace in both hexadecimal and character formats, while –trace-ascii stores only characters:
$ curl --trace-ascii trace.log http://www.example.com
<!doctype html>
...
$ cat trace.log
== Info: Trying ...
== Info: Connected to (nil) (....) port .... (#0)
=> Send header, 131 bytes (0x83)
0000: GET http://www.example.com/ HTTP/1.1
0026: Host: www.example.comThis option is useful when we want to debug and resolve connection issues.
5.3. Send an HTTP Post Request
There are plenty of options for sending data with curl and the POST method. We can send data like a browser submitting a form using the –data or -d option:
$ curl --data "param1=1¶m2=2" http://www.example.comAs can be seen, the value of the –data option is the data payload of the POST request.
Thus, the request sent with the above command is correctly formatted:
POST / HTTP/1.1
Host: localhost:8001
User-Agent: curl/7.81.0
Accept: */*
Content-Length: 17
Content-Type: application/x-www-form-urlencoded
param1=1¶m2=2Furthermore, we may use a filename with the –data option by placing the @ character as a prefix before said filename. In this case, the file should contain a correctly formatted data payload that we want to send:
$ cat '{"param1":"1","param2":"2"}' > request.json
$ curl --data @request.json http://www.example.comThe request was sent in the proper format:
Host: localhost:8001
User-Agent: curl/7.81.0
Accept: */*
Content-Length: 27
Content-Type: application/x-www-form-urlencoded
{"param1":"1","param2":"2"}So, in this example, we save some JSON data in the request.json file and provide the filename with the –data option.
Besides –data, we can use the –data-binary for sending binary data like images, and –data-raw if we don’t want to reserve the @ character for filenames.
5.4. Setting Headers
We can set an HTTP header using the –header or -H option:
$ curl -H 'Accept: text/html' http://www.example.comHere, we add the Accept HTTP header with a value of text/html.
5.5. Setting Cookies
To send a cookie with the Cookie HTTP header, we use the –cookie or -b option:
$ curl -b cookie1=1 http://www.example.comThe -b option adds the Cookie HTTP header:
Cookie: cookie1=1This option may be useful for sending HTTP requests to sites that store data on the client side.
6. Conclusion
In this article, we learned how to use the curl command.
First, we examined how to work with URLs and options. Then, we showed ways to interpret the command’s exit code. Finally, we demonstrated various practical examples.