curl

Command line tool and library for transferring data with URLs.

• https://curl.haxx.se/docs/httpscripting.html
• Download Curl

Usage examples

URL=https://example.com

HTTP GET

Headers only

curl --head "$URL"

Full server response – with web page code

curl "$URL"
curl "$URL" > temp.html

Using proxy

curl --proxy http://proxyservername.com:8080 "$URL"
curl --proxy-user proxyuser:proxypassword curl.haxx.se

Change user agent

curl --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL"

Basic authentication

curl --user name:password $URL

Send cookies with request

curl --cookie "name=Cookie name" "$URL"

Timeout

–connect-timeout: duration in seconds
–max-time: when performing multiple operations in a batch. This flag will set the duration for the whole operation – like downloading a big file.

curl --head --connect-timeout 5 $URL

curl --head --max-time 5 $URL

HTTP POST

curl --data "param1=value1&param2=value2" "$URL"

Encode POST data

curl --data-urlencode "param1=This is a test with spaces" "$URL"

HTTP POST with cookies

URL="https://${SERVERNAME}/index.jsp"
RESULTS="/root/Documents/results.html"
CONTENT_TYPE="Content-Type:application/x-www-form-urlencoded;charset=UTF-8;"
COOKIES="Cookie:JSESSIONID=${JSESSIONID};Token=${TOKEN}"
REQ1="sessionid=${USESSIONID}&csrftoken=${CSRFTOKEN}"

# -k : allow insecure certificates, ignore some certificate errors
# -b : Supply cookies with the request
# --header : Headers to supply with the request
# -i : Include HTTP headers in the output
curl -k -i -v -H $CONTENT_TYPE -H $COOKIES -d "${REQ1}" -X POST $URL > ${RESULTS}

Send multiple commands

# To send first a HEAD and then a GET
curl -I http://example.com --next http://example.com

# To first send a POST and then a GET
curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html 

Bash loop

for i in {1..1000}; do echo "Debug: $i"; curl --head "http://example.com?param=$i"; sleep 1; done