curl

由 datahunter 在六, 21/07/2012 - 14:48 發表

最後更新: 2021-12-28

Output to file(-o)

# -o output_file # 設定下載回來的東西寫到那裡去

curl -o output.file http://server/path/to/file.txt

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11274    0 11274    0     0  47239      0 --:--:-- --:--:-- --:--:-- 47369

* 仍有其他 output 的

想連 speed (progress meter) 那段也不看

-s, --silent Silent or quiet mode.

--write-out

-w, --write-out <format>

Make curl display information on stdout after a completed transfer.

All variables are specified as %{variable_name}

ie.

# --silent, --output
curl -w '%{http_code}\n' -s -o /dev/null https://datahunter.org

HTTP Login(-u)

curl -u username:password sftp://server/path/to/file.txt

curl -u username:password http://server/path/to/file.txt

curl http://username:passwd@server/path/to/file.txt

POST data by curl

Opts:

-d | --data <data>

Sends the specified data in a POST request to the HTTP server

Content-type "application/x-www-form-urlencoded"

It is the same as --data-ascii

--data-ascii <data>

This is just an alias for -d, --data

--data-binary <data>

post data purely binary

newlines and carriage returns are preserved and conversions are never done

If you start the data with the letter @, the rest should be a filename.

一般配合 -H "Content-Type: application/octet-stream" 使用

Default: "Content-Type: application/x-www-form-urlencoded"

--data-raw <data>

similarly to --data but without the special interpretation of the @ character.

--data-urlencode <data>

URL-encoding

Input(Data Source)

more than once "-d" => merged together with a separating "&"
@/path/to/file => filename to read the data from
The contents of the file must already be URL-encoded
- => read the data from stdin

Example

curl --data "param1=value1&param2=value2" http://example.com/resource.cgi

Specifying the Content-Type

i.e. JSON

curl -X POST -H "Content-Type: application/json" \
    -d '{"name": "postmaster", "email": "[email protected]"}' \
    https://example/contact

Emulate filled-in a form

-F | --form <name=content>

POST data using the Content-Type multipart/form-data (RFC 2388)

ie.

# data from file. 'profile', 'story' are the name of the form field

curl -F profile=@portrait.jpg https://example.com/upload.cgi

curl -F "story=<hugefile.txt" https://example.com/

# 改 upload filename

curl -F "file=@localfile;filename=nameinpost" example.com

# two text fields

curl -F name=John -F shoesize=11 https://example.co

# tell curl what Content-Type

curl -F "[email protected];type=text/html" example.com

-d vs -F

# 以下兩 cmd 係相等的

curl -d "log=admin&pwd=??" http://URL/wp-login.php

curl -F log=admin -F pwd=?? http://URL/wp-login.php

Header (-I vs -D)

Fetch Header 而已:

-I | --head # Fetch the HTTP-header only, 不是 show header response

curl -I datahunter.org

HTTP/1.1 200 OK
Date: Sun, 10 Feb 2013 15:59:43 GMT
Server: Apache/2.2.9 (Debian) DAV/2 PHP/5.2.6-1+lenny16 with ...
X-Powered-By: PHP/5.2.6-1+lenny16
Set-Cookie: SESS...=0a85...; expires=Tue, 05 Mar 2013 19:33:03 GMT; path=/
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Sun, 10 Feb 2013 15:59:43 GMT
Cache-Control: store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8

查看 Header 的方法 1

-i, --include

Include the HTTP response headers in the output.

查看 Header 的方法 2(保存 Header)

-D, --dump-header <filename>

i.e.

curl -D headers.txt http://localhost:8080

curl -D - http://localhost:8080

設定 Header:

curl -H 'Accept-Encoding: gzip' http://datahunter.org/ > download.gz

會見到以下一行:

Content-Encoding: gzip

P.S.

# no cache

curl -H "cache-Control: max-age=0" -k -D headers.txt https://datahunter.org/txt.ico > /dev/null; cat headers.txt

Cookie

-b, --cookie <data>

Pass the data to the HTTP server in the Cookie header.

-c, --cookie-jar <file name>

此方式會保儲及讀取 cookie

curl -c cookie.txt -I http://192.168.88.33:8081

Session Cookie:

It is supposedly the data previously received from the server in a "Set-Cookie:" line.

當 URL 有 redirect 時

-L / --location # curl redo the request on the new place

收到 server 回應 3xx 時, 會自動再去新地方

其他常用參數

直接執行 shell 時會用到

-s/--silent # Silent mode. Don't show progress meter or error messages.

-S, --show-error # When used with -s it makes curl show an error message if it fails.

Example:

curl -sS CRON_JOB_URL

curl -s SOME_SCRIPT_URL | bash

Buffer

-N, --no-buffer # Disables the buffering of the output stream.

lets curl emulate a filled-in form

-F/--form <form-field-name=content>

# This option can be used multiple times.

# "@" makes a file get attached

# "<" makes a text field and just get the contents for that text field from a file

# Content-Type: multipart/form-data

ie.

# send your password file to the server where 'password' is the name of the form-field

curl -F password=@/etc/passwd localhost

the name field of a file upload part by setting filename=

curl -F "file=@localfile;filename=nameinpost" localhost

-X/--request <command>

The specified request will be used instead of the method otherwise used (which defaults to GET).

SSL Version(-k)

Opts

-1, --tlsv1 # Forces curl to use TLS version 1
-3, --sslv3 # Forces curl to use SSL version 3
--tlsv1.3
-k, --insecure # This option explicitly allows curl to perform "insecure" SSL connections.

Certification:

Centos 7 的 CA 包: ca-certificates-2019.2.32-76.el7_7.noarch

Centos 6 的 CA 包: ca-certificates-2010.63-3.el6.noarch

Openssl certificate Path

openssl version -d

OPENSSLDIR: "/usr/share/ssl"        # C 6
OPENSSLDIR: "/etc/pki/tls"          # C 7

ls /etc/pki/tls

cert.pem  certs  dh2048_param.pem  dh512_param.pem  misc  openssl.cnf  private

curl default ssl version

CURL_SSLVERSION_DEFAULT - set preferred TLS/SSL version

PHP 內的 curl

<?
    $ch = curl_init();// set url
    curl_setopt($ch, CURLOPT_URL, "https://URL/ip.php");

    //return the as a string
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    // echo output string
    $output = curl_exec($ch);

    echo $output;

    // close curl resource to free up system resources
    curl_close($ch);
?>

bool curl_setopt ( resource $ch , int $option , mixed $value )

curl_setopt ($ch, CURLOPT_SSLVERSION, 6);

CURLOPT_SSLVERSION:

CURL_SSLVERSION_TLSv1_2 (6)

CA Certificates

Default Location: /usr/share/curl/curl-ca-bundle.crt

--cacert <CA certificate>

environment variable: CURL_CA_BUNDLE

Remark

If curl is built against the NSS SSL library (libnsspem.so) then

this option tells curl the nickname of the CA certificate to use within the NSS database defined by the environment variable

SSL_DIR (or by default /etc/pki/nssdb)

Limit speed

--limit-rate

Specify the maximum transfer rate you want curl to use

(for both downloads and uploads)

Default: bytes/second (Unit: k, m, g)

Timeout Setting

--connect-timeout <seconds>

Maximum time in seconds that you allow the connection to the server to take.

This only limits the connection phase, once curl has connected this option is of no more use.

-m, --max-time

Maximum time in seconds that you allow the whole operation to take.

This is useful for preventing your batch jobs from hanging for hours due to slow networks or links going down.

server:net_chk# /usr/local/bin/curl -m 2 https://1.2.3.4

curl: (28) Connection timed out after 2000 milliseconds

server:net_chk# echo $?

Specify one of IP for outgoing

curl --interface eth0 https://datahunter.org/ip.php

curl --interface eth0:1 https://datahunter.org/ip.php

cookie

load cookie

-b, --cookie <data>/file

It is supposedly the data previously received from the server in a "Set-Cookie:" line

The data should be in the format "NAME1=VALUE1; NAME2=VALUE2"

If no '=' symbol is used in the argument, it is instead treated as a filename to read previously stored cookie from.

To store cookies

-c, --cookie-jar option.

i.e.

curl -c MyCookie https://datahunter.org

curl 's cookies format

# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

datahunter.org  FALSE   /       Secure   ExpiresOn      NAME    VALUE

flag - A TRUE/FALSE value indicating if all machines within a given domain can access the variable.

path - The path within the domain that the variable is valid for.

secure - a secure connection with the domain is needed to access the variable

Expire 的時間

date --date='@1552296907'

SSLKEYLOGFILE environment

the SSLKEYLOGFILE environment variable can also be used with other software that uses NSS libraries,

for example Firefox.

Since curl version 7.57.0 the SSLKEYLOGFILE feature can also be enabled when built with GnuTLS, BoringSSL or OpenSSL.

Usage:

SSLKEYLOGFILE=ssl_log.txt curl https://URL

hosts file

curl -H "Host: datahunter.org" http://192.168.123.11/test.txt

Cookie

--cookie "USER_TOKEN=Yes"

Websocket

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

[Fix]

force the use of HTTP/1.1 with the argument --http1.1

Send requests to targets using HTTP/1.1. Supported when the request protocol is HTTP/1.1 or HTTP/2.

Troubleshoot

[1]

curl: (35) SSL connect error

------------------------------

測試

curl --tlsv1.2 https://xxxx

curl -3 https://xxxx

[2]

curl: (48) An unknown option was passed in to libcurl

ldd /usr/local/bin/curl

原來的有

libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f3d5476d000)

ldconfig

linux-vdso.so.1 (0x00007ffc9bfed000)
libcurl.so.4 => /usr/local/lib/libcurl.so.4 (0x00007f727d938000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f727d6cc000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f727d239000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f727d01f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f727ce02000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f727ca63000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f727c85f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f727dde7000)

byte-range

If the tool reports that it downloaded 100 bytes,

the media server correctly handled the byte-range request.

If it downloads the entire file, you may need to update the media server.

curl --range 0-99 https://URL/file.mp4 -o /dev/null

ETag

# Get ETag

curl -I URL

...
ETag: "XXXXXX"

# Check

curl -I URL --header 'If-None-Match: "XXXXXX"'

If the Etag matches, you should get a 304 response

HTTP/1.1 304 Not Modified

* 當 Server 不回應 ETag 時, 那也是 304

If-None-Match

For GET and HEAD methods

with a 200 status, only if it doesn't have an ETag matching the given ones.

P.S. 另有 "If-Modified-Since"

使用 curl 出信 (SMTP)

mail-by-curl.sh

#!/bin/bash

MAIL_FROM=user1@domain
MAIL_RCPT=user2@domain
EML=001.eml
SERVER='r.r.r.r'
PORT=25
curl smtp://$SERVER:$PORT -v \
  --mail-from $MAIL_FROM \
  --mail-rcpt $MAIL_RCPT \
  --upload-file $EML

Login

--user 'user@domain:password'

SSL

curl smtps://mail.domain:465

curl smtp://mail.domain:587 --ssl-reqd

--ssl-reqd: explicit STARTTLS connection to make sure SSL/TLS is used (FTP IMAP POP3 SMTP)

--ssl: Reverts to a non-secure connection if the server doesn't support SSL/TLS. (FTP IMAP POP3 SMTP)

EML

From: User1 <user1@domain>
To: User2 <user2@domain>
Subject: test msg
Date: Fri, 24 Dec 2021 07:22:09 +0000
Message-ID: 0001

This is my message

Log

...
> EHLO 001.eml
...
< 354 End data with <CR><LF>.<CR><LF>
} [145 bytes data]
* We are completely uploaded and fine
< 250 2.0.0 Ok: queued as 1A571321969
100   145    0     0  100   145      0   1510 --:--:-- --:--:-- --:--:--  1510
* Connection #0 to host mail.domain left intact

curl 收信 (POP3, IMAP)

[POP3]

# Setting
Server='mail.example.com'
Login='user@domain:password'

# list

curl pop3://$Server/ --user $Login

# NofMsg Size
1 10033
2 843
3 892
...

# download message 1 (Download 完 mail 依然在 Server)

curl pop3://$Server/1 --user $Login -o 1.eml

# To delete message 1

curl -X DELE pop3://$Server/1 --user $Login -v -m5

Remark

-X, --request # request to send to server

* If this option is used several times, the last one will be used.

-m, --max-time <seconds>

Maximum time in seconds that you allow the whole operation to take.

...
* Operation timed out after 5000 milliseconds with 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 5000 milliseconds with 0 bytes received

[IMAP]

# Check UidValidity

-X 'EXAMINE INBOX'

# Get UID

-X 'fetch 1:* (UID FLAGS)'

# get count of message in INBOX
-X 'STATUS INBOX (MESSAGES)'

# get count of unread message in INBOX
-X 'STATUS INBOX (UNSEEN)'

# list all mailboxes/folders
-X 'LIST "" "*"'

# download message UID 1
curl imap://mail.server.tld/INBOX;UID=1'

Compressed

--compressed

(HTTP) Request a compressed response using one of the algorithms curl supports,
and save the uncompressed document.

If this option is used and the server sends an unsupported encoding,
curl will report an error.

Range

// Server -> Client
Accept-Ranges: bytes

// Client -> Server
Range: bytes=0-1023

i.e.

# 查看 Server 是否支援 range

curl -I https://datahunter.org/test.webp

...
accept-ranges: bytes

# Get 1KB

curl -H "Range: bytes=0-1023" https://datahunter.org/test.webp -o test.webp

ls -l test.webp

-rw-r--r-- 1 root root 1024 Dec 20 16:05 test.webp

增加新的回應
瀏覽次數： 1560

夢想家

curl

目錄

Output to file(-o)

--write-out

HTTP Login(-u)

POST data by curl

Header (-I vs -D)

Cookie

當 URL 有 redirect 時

其他常用參數

Buffer

lets curl emulate a filled-in form

-X/--request <command>

SSL Version(-k)

PHP 內的 curl

CA Certificates

Limit speed

Timeout Setting

Specify one of IP for outgoing

cookie

SSLKEYLOGFILE environment

hosts file

Cookie

Websocket

Troubleshoot

byte-range

ETag

使用 curl 出信 (SMTP)

curl 收信 (POP3, IMAP)

Compressed

Range