The S3 files storage service of AWS

This article introduces the S3 storage service of AWS.

The S3 files storage service of AWS

Buckets

On the page of the buckets list, you can:

  • display the bucket properties;

  • enter in a bucket,
  • create a new bucket,
  • modify the public access settings,

  • make empty a bucket,
  • delete a bucket.

Create a bucket

  • define the name and the region:

A bucket has a name, in a DNS-compliant format, and has a selected region.

The settings of a bucket can be copied from another bucket.

  • configure options:

All the versions of an object in the bucket can be kept: it’s versioning.

You can log :

  • requests for access to a bucket,
  • activity at the object-level, by using CloudTrail (not free).

The objects can be encrypted automatically.

You can lock objects in a bucket. For that, the versioning should be enabled.

You can monitor requests in a bucket by using CloudWatch (not free).

  • Set permissions:

Permissions can be granted after you create the bucket.

  • Review:

Edit public access settings

You can grant public access through ACLs and/or through Bucket policies.

If possible, you should block public access. If it’s not possible, you can set the permissions at the bucket level and at the objects level.

You can block public access granted through:

  • any ACLs;
  • only new ACLs;
  • any public bucket policies;
  • only new public bucket policies.

Bucket page

On the bucket page, you can:

  • upload an objet,
  • Create a folder,
  • download an objet,
  • change the encryption,
  • change the metadata,
  • restore a previous (???) version,
  • change the storage class, and so on.

Upload a file

You can upload file till 80 GB through the Web console. Beyond this limit, use the AWS CLI, AWS SDK, or Amazon S3 REST API.

You can set permissions for:

  • an user,
  • another account,
  • everybody (public permission).

The permission can be set:

  • on the bucket (read or write), through Bucket policies (JSON documents),
  • on the objects (list, create, overwrite, delete), through ACL.

The first way is prefered.

To set public permission on an object, public permission should be turned on at the bucket level. It’s not recommenced to set a public permission.

Properties

One uploaded, you can set properties :

  • The storage class:

Based on your use case, choose a storage class. It can be changed later.

Storage class Use case AZ Min storage duration Min billable object size Retrieval fees
Standard Frequently accessed data ≥ 3 AZ
Intelligent-Tiering Long-lived data with changing or unknown access patterns ≥ 3 AZs 30 days Per-object fees apply (for monitoring and automation fees)
Standard-IA Long-lived, infrequently accessed data ≥ 3 AZs 30 days 128 KB Per-GB fees apply
One Zone-IA Long-lived, infrequently accessed, non-critical data ≥ 1 AZ 30 days 128KB Per-GB fees apply
Glacier Archive data with retrieval times ranging from minutes to hours ≥ 3 AZs 90 days Per-GB fees apply
Glacier Deep Archive Archive data that rarely, if ever, needs to be accessed with retrieval times in hours ≥ 3 AZs 180 days Per-GB fees apply
Reduced Redundancy (Not recommended) Frequently accessed, non-critical data ≥ 3 AZs
  • Encryption:

Encryption at rest can be:

  • disabled,

  • done by a AWS S3 master-key,

  • done by a AWS KMS master-key.

  • Metadata:

Before to upload a file, you can define its metadata:

  • Cache-Control
  • Content-Disposition
  • Content-Encoding
  • Content-Language
  • Content-Type
  • Expires
  • Website-Redirect-Location
  • x-amz-meta-
  • Tag:

You can defined tags on an object.

Review:

Create folder

You can create a folder with:

  • a specified name (a suffix “/“ is added),
  • an ecryption or not: AES-256 (S3-Managed Keys - SSE-S3), AWS-KMS (AWS KMS-Managed Keys - SSE-KMS).

Bucket page

Overview

This page lists the content of the bucket.

Properties

Properties displays general, permissions and management properties.

  • Versioning:

You can enable versioning but not disable it. You can just suspend it, but it preserves the existing versions.

  • Server access logging:

Access log records provide details about access requests.

  • Static website hosting:

You can host a static website on S3. It will use an endpoint similar to http://mybucket.s3-website-us-west-2.amazonaws.com.

You can redirect requests to a bucket or domain to another bucket or domain.

  • Object-level logging:

CloudTrail allow to record the object-level API activity (additional cost). You have to select:

  • trails from the same region,
  • events : read (to log read APIs such as GetObject), write (to log write APIs such as PutObject).

  • Default encryption:

Objects can be automatically encrypted:

  • AES-256 (SSE-S3 - S3-Managed Keys),
  • KMS (SSE-KMS).

  • Object lock:

When creating a bucket, you can enable locking objects to prevent to delete it.

  • Tags:

To track project costs, use tags.

  • Transfer acceleration:

You can enable faster data transfers by using an accelerated endpoint (in the format mybucket.s3-accelerate.amazonaws.com), but it has a cost. You can’t disable it, you can just suspend it.

  • Events:

You can receive notifications when specific events occur in a bucket.

  • Requester pays:

You can make pay the requester for data transfers instead of the bucket owner. But in this case, anonymous access is disabled.

Permissions

  • Block public access (bucket settings):

You can block public access.

  • Access Control List:

For each operation (list objects, write objects, read bucket, write bucket), you define the ACL for:

  • the access for your AWS account root user,

  • the access for other AWS accounts,

  • the public access (for everyone),

  • the S3 log delivery group.

  • Bucket Policy:

A bucket policy editor allow to enter the json file of a policy.

  • CORS configuration:

A CORS configuration editor allows to manage CORS configurations.

Management

Lifecycle

To manage object’s lifecycle:

  • define lifecycle rules: each rule defines filters to limit scope to prefix/tags;

  • define the transition for the current or the previous version to: Standard-IA, Intelligent-Tiering, One Zone-IA, Glacier, Glacier Deep Archive after a specified number of days (per-request fees);

  • configure the expiration (versioning should be enabled):
    • make the current version a previous version after a specified number of days from the date creation;
    • permanent delete the previous version after a specified number of days from the date creation.

  • ??? If all the previous versions of an object expire after deleting the object, the expired object delete marker is retained. But it has a cost. You can disable this behavior by cleaning up expired object delete markers.
  • You can automatically delete the parts of incomplete multipart uploads after a given number of days.

Review:

Replication

You can automatically and asynchronously copy objects across buckets:

  • in different regions (CRR - Cross region Replication),
  • in a same region (SRR - Same Region replication).

Replication requires versioning enabled.

Delete marker aren’t replicated.

You have provide:

  • the source (an entire bucket, tags or prefix),
    • You can replicate objects encrypted with AWS KMS.

  • the destination (a new bucket or an existing one in this account or in another account),
    • You can change the storage class for the replicated object.
    • You can change the object ownership to the destination bucket owner.

  • the options: an IAM role and a rule name.
    • The replication rule can be disabled.

Review this operation:

Analytics

Storage class analysis analyzes the access patterns and suggests an age for transition of objects.

You can specify:

  • an entire bucket, a prefix, or tags,
  • filters,

  • the destination bucket (in this account or another one) for the export data with a prefix (optional).

Metrics

You can get metrics from CloudWatch about buckets usage (additional cost):

  • storage: total bucket size (bytes / day), number of ojects (count / day);
  • Request,
  • Data transfer.

You can select buckets by filter baed on a prefix or tags.

Inventory

You can receive inventories daily or weekly for a entire bucket or based on a prefix and filter the version (all versions or current version only).

Inventories are reports of objects information:

  • Size,
  • Last modified date,
  • Storage class,
  • Etag,
  • Multipart upload,
  • Replication status,
  • Encryption status,
  • Retention mode,
  • Retain until date,
  • Legal hold status,
  • Encryption algorithm.

List of the objects of a bucket

From the bucket content list, you can get the overview, the properties and the permissions of each file.

You can download or copy the URL of the object, or you can search data with Select From.

When you select an object, the details of the object are displayed:

  • an overview,
  • the properties,
  • the permissions,
  • create select from.

Overview

You can open, download, copy the URL of the object or you can make it public.

Properties

The properties page allows to configure:

  • the storage class, based on frequency of access;

  • the encryption (none, AES-256 or AWS-KMS);

  • the metadata:

  • the tags, to search, organize the objects:

  • the object lock to prevent an object from being deleted:

Permissions

You can specify permissions to:

  • read the object,
  • read the ACL of the object,
  • edit the ACL of the object.

Permissions can be set for:

  • the object owner,
  • other AWS accounts,
  • the public access.

Select From

S3 Select allows to extract records from a CSV, JSON or Parquet file using SQL.

The files can be :

  • compressed or not,
  • encrypted or not,
  • 128 MB file size maximum and 40 MB extraction size maximum from the console (for higher size, use the API).

# , ,

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×