Diving Deep with curl: Your Ubuntu Command-Line Web Swiss Army Knife

 

Diving Deep with curl: Your Ubuntu Command-Line Web Swiss Army Knife

Introduction

If you're an Ubuntu user who spends any time in the terminal, you've likely encountered the curl command. It's far more than just a simple way to download files; it's a powerful tool for interacting with web servers, debugging APIs, and automating tasks. Let's explore some of the most useful curl features for your Ubuntu system.

What is curl?

curl stands for "Client URL." It's a command-line tool that allows you to transfer data with URLs. Think of it as a versatile web browser that operates entirely from your terminal. It supports a wide range of protocols, including HTTP, HTTPS, FTP, and more.

Getting Started

If curl isn't already installed on your Ubuntu system (though it usually is), you can install it using:

Bash

sudo apt update
sudo apt install curl

Basic Usage: Downloading Files

The simplest use case is downloading a file. To download a file from a URL, use:

Bash

curl <URL>

For example, to download the Ubuntu manual:

Bash

curl https://help.ubuntu.com/stable/ubuntu-help/index.html > ubuntu_manual.html

The > operator redirects the output to a file named ubuntu_manual.html.

Saving with the Original Filename

If you want to save the file with its original filename, use the -O (uppercase O) option:

Bash

curl -O <URL>

Example:

Bash

curl -O https://www.example.com/myfile.zip

Following Redirects

Sometimes, a URL will redirect you to another location. To follow these redirects, use the -L option:

Bash

curl -L <URL>

Getting HTTP Headers

To see the HTTP headers returned by a server, use the -I (uppercase i) option:

Bash

curl -I <URL>

This is useful for debugging and checking server responses.

Making POST Requests

curl isn't just for downloading. You can also send data to web servers using POST requests. Use the -X option to specify the request method and the -d option to send data:

Bash

curl -X POST -d "name=John&age=30" <URL>

Or for JSON data:

Bash

curl -X POST -H "Content-Type: application/json" -d '{"name": "John", "age": 30}' <URL>

The -H option sets HTTP headers.

Authentication

If a server requires authentication, you can use the -u option to provide a username and password:

Bash

curl -u username:password <URL>

Or for basic auth

Bash

curl --user 'username:password' <URL>

Using Proxies

If you need to use a proxy server, use the -x option:

Bash

curl -x http://proxy.example.com:8080 <URL>

Verbose Output

For more detailed information about the curl operation, use the -v option:

Bash

curl -v <URL>

This will show you the request and response headers, as well as other debugging information.

Downloading Multiple Files

You can download multiple files at once by specifying multiple URLs:

Bash

curl -O <URL1> -O <URL2> -O <URL3>

Using curl in Scripts

curl is incredibly useful in shell scripts for automating tasks. For example, you can use it to check the status of a website, download files, or interact with APIs.

Example Script Snippet

Bash

#!/bin/bash

status=$(curl -s -o /dev/null -w "%{http_code}" 
    https://www.example.com)

if [ "$status" -eq 200 ]; then
  echo "Website is up!"
else
  echo "Website is down or returned status code: $status"
fi

Key Options to Remember

Option	Description
-O	Save the output to a file with the original filename.
-L	Follow redirects.
-I	Get HTTP headers.
-X	Specify the request method (e.g., POST, PUT, DELETE).
-d	Send data in a POST request.
-H	Set HTTP headers.
-u	Provide authentication credentials.
-x	Use a proxy server.
-v	Verbose output.
-s	Silent mode (no progress bar or errors).
-w	Format information about the transfer.

Conclusion

curl is a versatile and powerful tool that every Ubuntu user should be familiar with. Whether you're downloading files, debugging web applications, or automating tasks, curl can simplify your workflow. Experiment with the various options and discover how it can enhance your command-line experience.

Need Ubuntu Expertise?

If you need help with your Ubuntu projects or have any questions, feel free to reach out to us!

Email us at: info@pacificw.com

Image:  Gemini