Skip to main content

S3 Storage objects caching

Service - CDN for Buckets

CDN for buckets addresses the following issues:

  • Reduced costs for S3 provider services by decreasing the number of requests to original objects in storage.
  • Improved loading speed for objects.

Setting up​

The following S3 provider details are required to enable CDN for bucket objects:

  • Provider type (AWS, Google, etc.)
  • Base URL of the S3 provider, e.g., https://s3.eu-central-1.amazonaws.com/
  • Bucket name, e.g., bucket-1

Review your provider setting before setting up

Example 1. Original bucket is located at https://example-bucket1.s3.us-west-2.amazonaws.com/:

  • Provider domain (provider_host): s3.us-west-2.amazonaws.com
  • Provider protocol: https
  • Provider port: 443
  • Bucket name: example-bucket1

Example 2. Original bucket is located at http://storage.googleapis.com/example-bucket2/:

  • Provider domain (provider_host): storage.googleapis.com
  • Provider protocol: http
  • Provider port: 80
  • Bucket name: example-bucket2

S3 Providers​

Arviol supports caching for objects from major S3 providers (s3_provider):

To request adding a new provider, contact support.

S3 Resources​

The base object for connecting CDN for S3 storage objects is s3_resource.
S3 resource serves as foundation for buckets that will be added by client. It's required to configure interaction between client's original buckets and buckets created in the system.

S3 Buckets​

After creating a resource, you can create buckets by specifying the particular resource. Each resource can have up to 20 buckets.
Each bucket can have up to 20 HTTP headers that steer CDN behavior.

headers are response headers the CDN returns to end users when serving bucket objects. Use them to define caching policies (Cache-Control), enable hardening measures (Strict-Transport-Security, X-Content-Type-Options), or attach diagnostic tags.

request_headers are headers the CDN forwards to the origin while fetching the original object. They are useful for passing routing hints, A/B test identifiers, or other operational metadata expected by the client’s backend.

CORS headers (cors_headers) are configured separately. CORS header names and quantity are predefined. Learn more about CORS.

Cache management​

Each bucket has cache purge methods available. These methods are used when actually replacing existing storage objects without changing their paths. When purging cache, objects are removed from caching servers and repopulated on subsequent first requests from bucket users. Partial cache cleanup is possible only for objects whose keys match the specified patterns. Cache purge can be called no more than once every 30 minutes.

Access restriction​

S3 buckets and resources support access restriction from specific countries.

Signed URLs​

Signed URLs are temporary unique links that allow access to objects in an S3 bucket without direct public access. They protect content and enable granting time-limited access to download or upload objects.

A secret key is used to generate a signed URL, which must be securely stored both in our system and by the client. The client generates signed URLs independently. Below is Python 3 code for a function that generates signed URLs.

import binascii
import hashlib
import hmac
import time
from urllib.parse import urlparse

def sign_url(url: str,
             secret_key: str,
             acl: str = "*",
             lifetime: int = 31_536_000,
             ) -> str:
    """
    Generate a signed URL with an expiration token for access control

    Args:
        url (str): URL to be signed
        secret_key (str): Secret key for signing (UTF-8 string)
        acl (str): Access control list, defaults to "*" (all paths)
        lifetime (int): Token lifetime in seconds, defaults to 1 year

    Returns:
        str: Signed URL with token appended as query parameter

    Example usage:
    >>> sign_url(
    ...     url="https://example.com/private/example.jpg",
    ...     secret_key="secret",
    ...     acl="/private/*",
    ...     lifetime=60 * 60 * 7,
    ... )
    'https://example.com/private/example.jpg?token=exp=1759879396~acl=/private/*~hmac=4fdb5e8bd60bbdeca1ddbeb93677de595a02d424e8211a3c0fb17e3735950db4'
    """

    exp = int(time.time() + lifetime)
    token_params = f"exp={exp}~acl={acl}"
    key_hex = secret_key.encode("utf-8").hex()

    token_hmac = hmac.new(
        key=binascii.a2b_hex(key_hex.encode()),
        msg=token_params.encode(),
        digestmod=hashlib.sha256,
    )
    token_digest = token_hmac.hexdigest()
    token = f"{token_params}~hmac={token_digest}"

    if urlparse(url).query:
        return f"{url}&token={token}"

    return f"{url}?token={token}"

The lifetime parameter specifies the validity period of the URL in seconds. The acl parameter is a path pattern specifying which resources the URL grants access to, relative to the base site URL. Examples of acl values:

  • * — all files (default)
  • /media/private_* — all files in the media folder with prefix private_
  • /media/* — all files in the media folder and its subfolders
  • /media/example.mp4 — only the file example.mp4 in the media folder
note

The acl for an S3 bucket must include the bucket name. For example, to restrict access to the bucket https://s403.s3-clients.cdn.itglobal.com/my-bucket, the acl value should be /my-bucket/*

If the client suspects the secret key has been compromised, they can change the key. In that case, all previously issued signed URLs will become invalid, ensuring secure access.

Object hierarchy​

Final hierarchy for static website caching management:

{
    "s3_resource": {
        "provider": "aws",
        "buckets": [
            {
                "headers": [
                    {
                        "key": "value"
                    }
                ],
                "request_headers": [
                    {
                        "key": "X-Origin-Feature",
                        "value": "image-optimizer"
                    }
                ],
                "cors_headers": {
                    "access_control_allow_origin": [
                        "*"
                    ],
                    "access_control_expose_headers": [
                        "*"
                    ],
                    "access_control_max_age": 600,
                    "access_control_allow_credentials": true,
                    "access_control_allow_methods": [
                        "DELETE"
                    ],
                    "access_control_allow_headers": [
                        "*"
                    ]
                }
            }
        ],
        "other_params": []
    }
}