Skip to content

Unreal Cloud DDC Infra Module

Unreal Cloud Derived Data Cache (source code) is a caching system that stores additional data required to use assets, such as compiled shaders. This allows the engine to quickly retrieve this data instead of having to regenerate it, saving time and disk space for the development team. For distributed teams, a cloud-hosted DDC enables efficient collaboration by ensuring all team members have access to the same cached data regardless of their location. This module deploys the core infrastructure for Unreal Engine's Cloud Derived Data Cache (DDC) on AWS. It creates a scalable, secure, and high-performance environment that optimizes asset processing and distribution throughout your game development pipeline, reducing build times and improving team collaboration.

The Unreal Cloud Derived Data Cache (DDC) infrastructure module implements Epic's recommended architecture using ScyllaDB, a high-performance Cassandra-compatible database. This module provisions the following AWS resources:

  1. ScyllaDB Database Layer:

    • Deployed on EC2 instances
    • Supports both single-node and multi-node cluster configurations
    • Optimized for high-throughput DDC operations
    • Configured with AWS Systems Manager Session Manager to provide secure shell access without requiring SSH or bastion hosts

  2. ScyllaDB Monitoring Stack:

    • Deployed on an EC2 instance
    • Uses Prometheus for metrics collection, Alertmanager for handling alerts, and Grafana for visualization
    • Creates a Application Load Balancer for accessing the Grafana UI for real-time insights into ScyllaDB node performance

  3. Amazon EKS Cluster with specialized node groups:

    • System node group: Handles core Kubernetes components and system workloads
    • NVME node group: Optimized for high-performance storage operations
    • Worker node group: Manages regional data replication and distribution
    • Configured with AWS Systems Manager Session Manager to provide secure shell access without requiring SSH or bastion hosts

  4. S3 Bucket:

    • Provides durable storage for cached assets
    • Enables cross-region asset availability
    • Serves as a persistent backup layer

Deployment Architecture


Unreal Engine Cloud DDC Infra Module Architecture


Prerequisites

Network Infrastructure Requirements

At a minimum, the Cloud DDC Module requires a Virtual Private Cloud (VPC) with a specific subnet configuration. The suggested configuration includes:

  • 2 public subnets
  • 2 private subnets
  • Coverage across 2 Availability Zones
  • An S3 interface endpoint

This architecture ensures high availability and secure communication patterns for your DDC infrastructure.


Configuring Node Groups and ScyllaDB Deployment

The footprint of your Cloud DDC deployment can be configured through 2 variables:


EKS Node Group Configuration: eks_node_group_subnets

The eks_node_group_subnets variable defines the subnet distribution for your EKS node groups. Each specified subnet serves as a potential target for node placement, providing granular control over the geographical distribution of your EKS infrastructure. Adding more subnets to this configuration increases deployment flexibility and enables broader availability zone coverage for your workloads at the cost of increased network complexity and potential inter-AZ data transfer charges.


ScyllaDB Instance Distribution: scylla_subnets

The scylla_subnets variable determines the deployment topology of your ScyllaDB instances. Each specified subnet receives a dedicated ScyllaDB instance, with multiple subnet configurations automatically establishing a distributed cluster architecture. Configurations of two or more subnets enable high availability and data resilience through native ScyllaDB clustering at the cost of increased infrastructure complexity and proportionally higher operational expenses.

Requirements

Name Version
terraform >= 1.5
aws >= 5.38
tls >= 4.0.5

Providers

Name Version
aws >= 5.38
tls >= 4.0.5

Modules

No modules.

Resources

Name Type
aws_cloudwatch_log_group.unreal_cluster_cloudwatch resource
aws_eks_cluster.unreal_cloud_ddc_eks_cluster resource
aws_eks_identity_provider_config.eks_cluster_oidc_association resource
aws_eks_node_group.nvme_node_group resource
aws_eks_node_group.system_node_group resource
aws_eks_node_group.worker_node_group resource
aws_iam_instance_profile.scylla_instance_profile resource
aws_iam_openid_connect_provider.unreal_cloud_ddc_oidc_provider resource
aws_iam_role.eks_cluster_role resource
aws_iam_role.monitoring_node_group_role resource
aws_iam_role.nvme_node_group_role resource
aws_iam_role.scylla_role resource
aws_iam_role.worker_node_group_role resource
aws_instance.scylla_ec2_instance resource
aws_launch_template.nvme_launch_template resource
aws_launch_template.system_launch_template resource
aws_launch_template.worker_launch_template resource
aws_route53_record.scylla_records resource
aws_route53_zone.scylla_zone resource
aws_s3_bucket.unreal_ddc_logging_s3_bucket resource
aws_s3_bucket.unreal_ddc_s3_bucket resource
aws_s3_bucket_logging.unreal-log-s3-log resource
aws_s3_bucket_logging.unreal-s3-log resource
aws_s3_bucket_public_access_block.unreal_ddc_log_s3_acls resource
aws_s3_bucket_public_access_block.unreal_ddc_s3_acls resource
aws_s3_bucket_server_side_encryption_configuration.unreal-s3-bucket resource
aws_s3_bucket_server_side_encryption_configuration.unreal-s3-logging-bucket resource
aws_security_group.nvme_security_group resource
aws_security_group.scylla_security_group resource
aws_security_group.system_security_group resource
aws_security_group.worker_security_group resource
aws_security_group_rule.peer_cidr_blocks_ingress_sg_rules resource
aws_security_group_rule.peer_cidr_blocks_scylla_egress_sg_rules resource
aws_security_group_rule.scylla_to_nvme_group_egress_sg_rules resource
aws_security_group_rule.scylla_to_nvme_group_ingress_sg_rules resource
aws_security_group_rule.scylla_to_worker_group_egress_sg_rules resource
aws_security_group_rule.scylla_to_worker_group_ingress_sg_rules resource
aws_security_group_rule.self_ingress_sg_rules resource
aws_security_group_rule.self_scylla_egress_sg_rules resource
aws_security_group_rule.ssm_egress_sg_rules resource
aws_ssm_association.scylla_config_association resource
aws_ssm_document.config_scylla resource
aws_ami.scylla_ami data source
aws_ssm_parameter.eks_ami_latest_release data source
tls_certificate.eks_tls_certificate data source

Inputs

Name Description Type Default Required
eks_cluster_access_cidr List of the CIDR Ranges you want to grant public access to the EKS Cluster. list(string) n/a yes
name Unreal Cloud DDC Workload Name string "unreal-cloud-ddc" no
nvme_managed_node_desired_size Desired number of nvme managed node group instances number 2 no
nvme_managed_node_instance_type Nvme managed node group instance type string "i3en.xlarge" no
nvme_managed_node_max_size Max number of nvme managed node group instances number 2 no
peer_cidr_blocks The peered cidr blocks you want your vpc to communicate with if you have a multi region ddc. list(string) [] no
private_subnets Private subnets you want scylla and the worker nodes to be installed into. list(string) [] no
scylla_ami_name Name of the Scylla AMI to be used to get the AMI ID string "ScyllaDB 6.0.1" no
scylla_architecture The chip architecture to use when finding the scylla image. Valid string "x86_64" no
scylla_db_storage Size of gp3 ebs volumes attached to Scylla DBs number 100 no
scylla_db_throughput Throughput of gp3 ebs volumes attached to Scylla DBs number 200 no
scylla_dns The local private dns name that you want Scylla to be queryable on. string null no
scylla_instance_type The type and size of the Scylla instance. string "i4i.2xlarge" no
scylla_private_subnets The subnets you want Scylla to be installed into. Can repeat subnet ids to install into the same subnet/az. This will also determine how many Scylla instances are deployed. list(string) [] no
system_managed_node_desired_size Desired number of monitoring managed node group instances. number 1 no
system_managed_node_instance_type Monitoring managed node group instance type. string "m5.large" no
system_managed_node_max_size Max number of monitoring managed node group instances. number 2 no
vpc_id String for VPC ID string n/a yes
worker_managed_node_desired_size Desired number of worker managed node group instances. number 1 no
worker_managed_node_instance_type Worker managed node group instance type. string "c5.xlarge" no
worker_managed_node_max_size Max number of worker managed node group instances. number 1 no

Outputs

Name Description
cluster_arn n/a
cluster_certificate_authority_data n/a
cluster_endpoint n/a
cluster_name n/a
oidc_provider_arn n/a
oidc_provider_identity n/a
s3_bucket_id n/a

Requirements

Name Version
terraform >= 1.10.3
aws >=5.89.0
random 3.5.1
tls >= 4.0.6

Providers

Name Version
aws 5.99.1
random 3.5.1
tls 4.1.0

Modules

No modules.

Resources

Name Type
aws_cloudwatch_log_group.unreal_cluster_cloudwatch resource
aws_eks_cluster.unreal_cloud_ddc_eks_cluster resource
aws_eks_node_group.nvme_node_group resource
aws_eks_node_group.system_node_group resource
aws_eks_node_group.worker_node_group resource
aws_iam_instance_profile.scylla_instance_profile resource
aws_iam_instance_profile.scylla_monitoring_profile resource
aws_iam_openid_connect_provider.unreal_cloud_ddc_oidc_provider resource
aws_iam_role.eks_cluster_role resource
aws_iam_role.nvme_node_group_role resource
aws_iam_role.scylla_monitoring_role resource
aws_iam_role.scylla_role resource
aws_iam_role.system_node_group_role resource
aws_iam_role.worker_node_group_role resource
aws_iam_role_policy.scylla_monitoring_policy resource
aws_iam_role_policy_attachments_exclusive.eks_cluster_policy_attachement resource
aws_iam_role_policy_attachments_exclusive.nvme_policy_attachement resource
aws_iam_role_policy_attachments_exclusive.scylla_policy_attachement resource
aws_iam_role_policy_attachments_exclusive.system_policy_attachement resource
aws_iam_role_policy_attachments_exclusive.worker_policy_attachement resource
aws_instance.scylla_ec2_instance_other_nodes resource
aws_instance.scylla_ec2_instance_seed resource
aws_instance.scylla_monitoring resource
aws_launch_template.nvme_launch_template resource
aws_launch_template.system_launch_template resource
aws_launch_template.worker_launch_template resource
aws_lb.scylla_monitoring_alb resource
aws_lb_listener.scylla_monitoring_listener resource
aws_lb_target_group.scylla_monitoring_alb_target_group resource
aws_lb_target_group_attachment.scylla_monitoring resource
aws_s3_bucket.scylla_monitoring_lb_access_logs_bucket resource
aws_s3_bucket.unreal_ddc_s3_bucket resource
aws_s3_bucket_lifecycle_configuration.access_logs_bucket_lifecycle_configuration resource
aws_s3_bucket_policy.alb_access_logs_bucket_policy resource
aws_s3_bucket_public_access_block.access_logs_bucket_public_block resource
aws_s3_bucket_public_access_block.unreal_ddc_s3_acls resource
aws_security_group.cluster_security_group resource
aws_security_group.nvme_security_group resource
aws_security_group.scylla_monitoring_lb_sg resource
aws_security_group.scylla_monitoring_sg resource
aws_security_group.scylla_security_group resource
aws_security_group.system_security_group resource
aws_security_group.worker_security_group resource
aws_vpc_security_group_egress_rule.cluster_egress_sg_rule resource
aws_vpc_security_group_egress_rule.nvme_egress_sg_rules resource
aws_vpc_security_group_egress_rule.scylla_monitoring_lb_sg_egress_rule resource
aws_vpc_security_group_egress_rule.scylla_monitoring_sg_egress_rule resource
aws_vpc_security_group_egress_rule.self_scylla_egress_sg_rules resource
aws_vpc_security_group_egress_rule.ssm_egress_sg_rules resource
aws_vpc_security_group_egress_rule.system_egress_sg_rules resource
aws_vpc_security_group_egress_rule.worker_egress_sg_rules resource
aws_vpc_security_group_ingress_rule.cluster_lb_ingress_sg_rule resource
aws_vpc_security_group_ingress_rule.scylla_monitoring_ingress_node_exporter resource
aws_vpc_security_group_ingress_rule.scylla_monitoring_ingress_prometheus resource
aws_vpc_security_group_ingress_rule.scylla_monitoring_lb_monitoring resource
aws_vpc_security_group_ingress_rule.self_ingress_cluster_sg_rule resource
aws_vpc_security_group_ingress_rule.self_ingress_sg_rules resource
random_string.scylla_monitoring_lb_access_logs_bucket_suffix resource
aws_ami.amazon_linux data source
aws_ami.scylla_ami data source
aws_elb_service_account.main data source
aws_iam_policy_document.access_logs_bucket_alb_write data source
aws_iam_policy_document.scylla_monitoring_assume_role data source
aws_iam_policy_document.scylla_monitoring_policy_doc data source
tls_certificate.eks_tls_certificate data source

Inputs

Name Description Type Default Required
alb_certificate_arn The ARN of the certificate to use on the ALB string null no
create_application_load_balancer Whether to create an application load balancer for the Scylla monitoring dashboard. bool true no
create_scylla_monitoring_stack Whether to create the Scylla monitoring stack bool true no
debug Enable debug mode bool false no
eks_cluster_cloudwatch_log_group_prefix Prefix to be used for the EKS cluster CloudWatch log group. string "/aws/eks/unreal-cloud-ddc/cluster" no
eks_cluster_logging_types List of EKS cluster log types to be enabled. list(string) 
[
  "api",
  "audit",
  "authenticator",
  "controllerManager",
  "scheduler"
]
 no
eks_cluster_private_access Allows private access of the EKS Control Plane from subnets attached to EKS Cluster bool true no
eks_cluster_public_access Allows public access of EKS Control Plane should be used with bool false no
eks_cluster_public_endpoint_access_cidr List of the CIDR Ranges you want to grant public access to the EKS Cluster's public endpoint. list(string) [] no
eks_node_group_subnets A list of subnets ids you want the EKS nodes to be installed into. Private subnets are strongly recommended. list(string) [] no
enable_scylla_monitoring_lb_access_logs Whether to enable access logs for the Scylla monitoring load balancer. bool false no
enable_scylla_monitoring_lb_deletion_protection Whether to enable deletion protection for the Scylla monitoring load balancer. bool false no
environment The current environment (e.g. dev, prod, etc.) string "dev" no
existing_security_groups List of existing security groups to add to the monitoring and Unreal DDC load balancers list(string) [] no
internal_facing_application_load_balancer Whether the application load balancer should be internal-facing. bool false no
kubernetes_version Kubernetes version to be used by the EKS cluster. string "1.31" no
monitoring_application_load_balancer_subnets The subnets in which the ALB will be deployed list(string) null no
name Unreal Cloud DDC Workload Name string "unreal-cloud-ddc" no
nvme_managed_node_desired_size Desired number of nvme managed node group instances number 2 no
nvme_managed_node_instance_type Nvme managed node group instance type string "i3en.large" no
nvme_managed_node_max_size Max number of nvme managed node group instances number 2 no
nvme_managed_node_min_size Min number of nvme managed node group instances number 1 no
nvme_node_group_label Label applied to nvme node group. These will need to be matched in values for taints and tolerations for the worker pod definition. map(string) 
{
  "unreal-cloud-ddc/node-type": "nvme"
}
 no
project_prefix The project prefix for this workload. This is appended to the beginning of most resource names. string "cgd" no
region The AWS region to deploy to string "us-west-2" no
scylla_ami_name Name of the Scylla AMI to be used to get the AMI ID string "ScyllaDB 6.0.1" no
scylla_architecture The chip architecture to use when finding the scylla image. Valid string "x86_64" no
scylla_db_storage Size of gp3 ebs volumes attached to Scylla DBs number 100 no
scylla_db_throughput Throughput of gp3 ebs volumes attached to Scylla DBs number 200 no
scylla_instance_type The type and size of the Scylla instance. string "i4i.2xlarge" no
scylla_monitoring_instance_storage Size of gp3 ebs volumes in GB attached to Scylla monitoring instance number 20 no
scylla_monitoring_instance_type The type and size of the Scylla monitoring instance. string "t3.xlarge" no
scylla_monitoring_lb_access_logs_bucket Name of the S3 bucket to store the access logs for the Scylla monitoring load balancer. string null no
scylla_monitoring_lb_access_logs_prefix Prefix to use for the access logs for the Scylla monitoring load balancer. string null no
scylla_subnets A list of subnet IDs where Scylla will be deployed. Private subnets are strongly recommended. list(string) [] no
system_managed_node_desired_size Desired number of system managed node group instances. number 1 no
system_managed_node_instance_type Monitoring managed node group instance type. string "m5.large" no
system_managed_node_max_size Max number of system managed node group instances. number 2 no
system_managed_node_min_size Min number of system managed node group instances. number 1 no
system_node_group_label Label applied to system node group map(string) 
{
  "pool": "system-pool"
}
 no
tags Tags to apply to resources. map(any) 
{
  "IaC": "Terraform",
  "ModuleBy": "CGD-Toolkit",
  "ModuleName": "Unreal DDC"
}
 no
vpc_id String for VPC ID string n/a yes
worker_managed_node_desired_size Desired number of worker managed node group instances. number 1 no
worker_managed_node_instance_type Worker managed node group instance type. string "c5.large" no
worker_managed_node_max_size Max number of worker managed node group instances. number 1 no
worker_managed_node_min_size Min number of worker managed node group instances. number 0 no
worker_node_group_label Label applied to worker node group. These will need to be matched in values for taints and tolerations for the worker pod definition. map(string) 
{
  "unreal-cloud-ddc/node-type": "worker"
}
 no

Outputs

Name Description
cluster_arn ARN of the EKS Cluster
cluster_certificate_authority_data Public key for the EKS Cluster
cluster_endpoint EKS Cluster Endpoint
cluster_name Name of the EKS Cluster
external_alb_dns_name DNS endpoint of Application Load Balancer (ALB)
external_alb_zone_id Zone ID for internet facing load balancer
nvme_node_group_label Label for the NVME node group
oidc_provider_arn OIDC provider for the EKS Cluster
peer_security_group_id ID of the Peer Security Group
s3_bucket_id Bucket to be used for the Unreal Cloud DDC assets
scylla_ips IPs of the Scylla EC2 instances
system_node_group_label Label for the System node group
worker_node_group_label Label for the Worker node group