最後更新: 2022-05-13

目錄

• list (ls)
• upload & download (cp)
• delete (rm)
• Sync Content(sync)
• Improve the transfer performance

 

---

S3 CLI

 

List

# list bucket

aws s3 ls

2021-08-19 03:21:44 my-import-export-bucket

P.S.

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

# list content in bucket

aws s3 ls s3://my-import-export-bucket

2021-08-19 03:22:55 1063377408 c7.mini.ova

P.S.

fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Opts

• --human-readable      # Unit: Bytes
• --summarize              # CLI 尾列出 "Total Objects" & "Total Size"

Count files in bucket

aws s3 ls --recursive --human-readable --summarize s3://mybucket/

aws s3 ls --recursive | wc -l s3://mybucket/

P.S.

GUI: Amazon S3 > Buckets > YOUR_Bucket > Metircs     # 每隔一段時間採樣一次

Upload & Download

# Download File

aws s3 cp s3://mybucket/myfolder/file /backup

Completed 826.2 MiB/25.6 GiB (56.8 MiB/s) with 1 file(s) remaining

 * overwrite 時係沒有 confime 的

# Upload File

aws s3 cp test.txt s3://mybucket/myfolder

 * overwrite 時係沒有 confime 的

# Upload Folder

aws s3 cp myfolder --recursive s3://mybucket/myfolder

Delete

aws s3 rm s3://mybucket/myfolder/file

Doc

http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

 

---

Sync Content

 

Recursively copies new(新建立) or updated(時間不同, 並不是指新過) files from the source directory to the destination (no delete on dest)

Only creates folders in the destination if they contain one or more files.

Opt:

• --dryrun

i.e.

aws s3 sync --dryrun s3://MyBucket ./ > ../s3_sync.log

cat ../s3_sync.log

(dryrun) upload: files/test.txt to s3://MyBucket/files/test.txt
...

# Recursive Permission

# Command is performed on all files or objects under the specified directory or prefix.

# Only creates folders in the destination if they contain one or more files.

# To run the command aws s3 cp with the --recursive option, you need permission to

s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket

P.S.

An error occurred (AccessDenied) when missing "ListObjectsV2" permission

Note:

s3:ListBucket is the name of the permission that allows a user to list the objects in a bucket.

ListObjectsV2 is the name of the API call that lists the objects in a bucket.

"Action": [
    "s3:ListBucket",
],
"Resource": [
    "arn:aws:s3:::bucketname",
    "arn:aws:s3:::bucketname/*"
]

* Specify bucket resource ARN for the ListBucketVersions and 2 more actions.

ie.

# Local -> S3

aws s3 sync . s3://mybucket

# S3 -> Local

aws s3 sync s3://mybucket ./

download: s3://mybucket/test1.txt to ./test1.txt
download: s3://mybucket/test3.txt to ./test3.txt
download: s3://mybucket/test2.txt to ./test2.txt

 * updated(時間不同, 並不是指新過) files

touch test2.txt

aws s3 sync s3://mybucket ./

download: s3://mybucket/test2.txt to ./test2.txt

# Delete dest missing

# 刪除只存在於 dst 的 file (那 file 不存在於 src)

aws s3 sync . s3://mybucket --delete

# Exclude file

aws s3 sync ./myfolder s3://mybucket/myfolder --exclude *.tmp

# S3 to S3

aws s3 sync s3://mybucket s3://mybucket2

aws s3 sync s3://mybucket/myfolder s3://mybucket/myfolder2

# two buckets in different regions

aws s3 sync s3://my-us-west-2-bucket s3://my-us-east-1-bucket --source-region us-west-2 --region us-east-1

# Verify that the objects are copied

aws s3 ls --recursive --summarize s3://mybucket-SOURCE > bucket-contents-source.txt

 * Versioning 的檔案不 count 在內 !!

aws s3 ls --recursive --summarize s3://mybucket-TARGET > bucket-contents-target.txt

 

---

Improve the transfer performance

 

max_concurrent_requests

At any given time, multiple requests to Amazon S3 are in flight. (Default: 10)

config

aws configure set default.s3.max_concurrent_requests 15

cat ~/.aws/config

[default]
region = ap-east-1
s3 =
    max_concurrent_requests = 15