AWS S3 CLI

由 datahunter 在四, 05/05/2022 - 10:40 發表

最後更新: 2022-05-13

S3 CLI

List

# list bucket

aws s3 ls

2021-08-19 03:21:44 my-import-export-bucket

P.S.

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

# list content in bucket

aws s3 ls s3://my-import-export-bucket

2021-08-19 03:22:55 1063377408 c7.mini.ova

P.S.

fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Opts

--human-readable # Unit: Bytes
--summarize # CLI 尾列出 "Total Objects" & "Total Size"

Count files in bucket

aws s3 ls --recursive --human-readable --summarize s3://mybucket/

aws s3 ls --recursive | wc -l s3://mybucket/

P.S.

GUI: Amazon S3 > Buckets > YOUR_Bucket > Metircs # 每隔一段時間採樣一次

Upload & Download

# Download File

aws s3 cp s3://mybucket/myfolder/file /backup

Completed 826.2 MiB/25.6 GiB (56.8 MiB/s) with 1 file(s) remaining

* overwrite 時係沒有 confime 的

# Upload File

aws s3 cp test.txt s3://mybucket/myfolder

* overwrite 時係沒有 confime 的

# Upload Folder

aws s3 cp myfolder --recursive s3://mybucket/myfolder

aws s3 ls s3://mybucket/

                           PRE myfolder/

aws s3 ls s3://mybucket/myfolder

                           PRE myfolder/

aws s3 ls s3://mybucket/myfolder/

Notes

如果沒有加 Folder 名, 那就會抄 source folder 內的 file 上去

upload: myfolder/test1.txt to s3://mybucket/test1.txt

Delete

aws s3 rm s3://mybucket/myfolder/file

Doc

http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

Sync Content

Recursively copies new(新建立) or updated(時間不同, 並不是指新過) files from the source directory to the destination (no delete on dest)

Only creates folders in the destination if they contain one or more files.

Deletion of destination files that no longer exist in the source can be optionally enabled. ()

Opt:

--dryrun
--no-progress

i.e.

# 必須 log 到其他目錄, 因為在當前目錄 Sync

aws s3 sync --dryrun s3://MyBucket ./ > ../s3_test.log

cat ../s3_test.log

(dryrun) upload: files/test.txt to s3://MyBucket/files/test.txt
...

# 必須 --no-progress 否則 log 會亂了. (大量 5% ... 10% ...)

aws s3 sync --no-progress s3://MyBucket ./ > ../s3_dl.log

cat ../s3_dl.log

Recursive Permission(--recursive)

# Command is performed on all files or objects under the specified directory or prefix.

# Only creates folders in the destination if they contain one or more files.

# To run the command aws s3 cp with the --recursive option, you need permission to

s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket

P.S.

An error occurred (AccessDenied) when missing "ListObjectsV2" permission

Note:

s3:ListBucket is the name of the permission that allows a user to list the objects in a bucket.

ListObjectsV2 is the name of the API call that lists the objects in a bucket.

"Action": [
    "s3:ListBucket",
],
"Resource": [
    "arn:aws:s3:::bucketname",
    "arn:aws:s3:::bucketname/*"
]

* Specify bucket resource ARN for the ListBucketVersions and 2 more actions.

ie.

# Local -> S3

aws s3 sync . s3://mybucket

# S3 -> Local

aws s3 sync s3://mybucket ./

download: s3://mybucket/test1.txt to ./test1.txt
download: s3://mybucket/test3.txt to ./test3.txt
download: s3://mybucket/test2.txt to ./test2.txt

* updated(時間不同, 並不是指新過) files

touch test2.txt

aws s3 sync s3://mybucket ./

download: s3://mybucket/test2.txt to ./test2.txt

# Delete dest missing

# 刪除只存在於 dst 的 file (那 file 不存在於 src)

aws s3 sync . s3://mybucket --delete

# Exclude file

aws s3 sync ./myfolder s3://mybucket/myfolder --exclude *.tmp

# S3 to S3

aws s3 sync s3://mybucket s3://mybucket2

aws s3 sync s3://mybucket/myfolder s3://mybucket/myfolder2

# two buckets in different regions

aws s3 sync s3://my-us-west-2-bucket s3://my-us-east-1-bucket --source-region us-west-2 --region us-east-1

# Verify that the objects are copied

aws s3 ls --recursive --summarize s3://mybucket-SOURCE > bucket-contents-source.txt

* Versioning 的檔案不 count 在內 !!

aws s3 ls --recursive --summarize s3://mybucket-TARGET > bucket-contents-target.txt

Improve the transfer performance

max_concurrent_requests

At any given time, multiple requests to Amazon S3 are in flight. (Default: 10)

config

aws configure set default.s3.max_concurrent_requests 15

cat ~/.aws/config

[default]
region = ap-east-1
s3 =
    max_concurrent_requests = 15

metadata

Get metadata

# aws s3api head-object --bucket BUCKET_NAME --key OBJECT_KEY

aws s3api head-object --bucket mybucket --key files/test.pdf

{
    "AcceptRanges": "bytes",
    "LastModified": "2024-05-14T08:11:29+00:00",
    "ContentLength": 193811,
    "ETag": "\"ID\"",
    "VersionId": "ID",
    "ContentType": "application/pdf",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

LastModified

upload start timestamp

sync 與 metadata

--content-type (string)

Specify an explicit content type for this operation.
This value overrides any guessed mime types.

--content-encoding (string)
Specifies what content encodings have been applied to the object and
thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field.

IAM Permission

Permissions against the Bucket are separate to permissions against Objects within the Bucket.

Therefore, you must grant permissions to both.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllAccess",
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::mybucket",
                "arn:aws:s3:::mybucket/*"
            ]
        }
    ]
}

Notes

要保留 "Version": "2012-10-17", 否則有 Warning !!

ls 的測試

aws s3 ls s3://mybucket # 成功

aws s3 ls # 失敗

# 要 list 到 bucket 的話就要加入 ListAllMyBuckets 權限 (a list of all buckets owned by the sender of the request)

    "Statement": [
        {
            ...
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
       }
    ]

瀏覽次數： 602

夢想家

AWS S3 CLI

目錄

S3 CLI

Sync Content

Improve the transfer performance

metadata

sync 與 metadata

IAM Permission