Unreal Cloud DDC Single Region¶
The Unreal Cloud DDC Single Region is a comprehensive solution that leverages several AWS services to create a robust and efficient data caching system. It uses a well-designed Virtual Private Cloud (VPC) to ensure network isolation and security. The solution employs an Amazon Elastic Kubernetes Service (EKS) Cluster with Node Groups to manage and orchestrate containerized applications.
At the heart of the system is an instance of ScyllaDB, a high-performance NoSQL database, running on specially optimized Amazon EC2 instances. The Unreal Cloud Derived Data Cache Container is managed by Helm, a package manager for Kubernetes, and uses Amazon S3 for durable storage.
Predeployment - Set Up Github Content Repository Credentials¶
The Unreal Cloud DDC Intra Cluster module utilizes a pull through cache to access the Unreal Cloud DDC image. This requires a secret in Secrets Manager. The secret needs to be prefixed with ecr-pullthroughcache/. Additionally, the secret is required to be in the following format:
{
"username":"GITHUB-USER-NAME-PLACEHOLDER",
"access-token":"GITHUB-ACCESS-TOKEN-PLACEHOLDER"
}
Deployment¶
Once you've completed the prerequisites and set your variables, you can deploy the solution by running:
terraform apply
The deployment can take close to 30 minutes. Creating the EKS Node Groups and EKS Cluster take around 20 minutes to fully deploy.
EBS CSI driver identity mode (IRSA vs EKS Pod Identity)¶
The EBS CSI driver needs AWS credentials to manage EBS volumes for the cluster.
This sample supports two mechanisms, selected by the sample's root
ebs_csi_use_pod_identity variable (default true), which is passed through
to the intra-cluster module:
- EKS Pod Identity (recommended, the sample default). Uses the
pods.eks.amazonaws.comservice principal andeks-auth:AssumeRoleForPodIdentity. This is AWS's current recommended mechanism, needs no cluster OIDC provider for the driver, and installs theeks-pod-identity-agentadd-on automatically. It also works in accounts whose AWS Organization applies an SCP/RCP that deniessts:AssumeRoleWithWebIdentity(a common central-governance guardrail) — under IRSA that guardrail leaves theebs-csi-controllerpods inCrashLoopBackOffwithAccessDenied, and the add-on never becomesACTIVE. - IRSA (legacy). Set
ebs_csi_use_pod_identity = falseto use the classic IAM Roles for Service Accounts path (sts:AssumeRoleWithWebIdentityvia the cluster OIDC provider). Appropriate when Pod Identity is unavailable (for example Fargate or Windows node groups).
# terraform.tfvars — override if you need the legacy IRSA path
ebs_csi_use_pod_identity = false
EKS API endpoint access¶
Terraform installs Helm charts and EKS add-ons over the cluster's Kubernetes
API, so the machine running terraform apply must be able to reach the EKS API
endpoint for the entire apply.
By default this sample auto-detects your current public IP and allows only that
single /32 on the cluster's public endpoint. This only works when your
egress is a single, stable public IP. If you are behind a corporate NAT pool
or VPN, or you run Terraform from CI, your outbound traffic may leave from a
different IP than the one auto-detected (or the IP may change mid-apply). The
symptom is Kubernetes cluster unreachable: ... dial tcp ...:443: i/o timeout
partway through the apply, even though the cluster came up.
In that case set eks_public_access_cidrs explicitly to your egress CIDR(s):
# terraform.tfvars
eks_public_access_cidrs = ["203.0.113.10/32", "203.0.113.11/32"]
For production, prefer keeping the public endpoint closed and running Terraform
from inside the VPC (bastion, VPN, or in-VPC CI) so the API is reached over the
private endpoint (eks_cluster_private_access is already enabled) rather than
widening public access.
Reaching the API over the private endpoint via SSM¶
If your host cannot reach the public endpoint at all (corporate egress filters
it, even with the correct eks_public_access_cidrs), you can have Terraform
talk to the cluster's private endpoint through an SSM port-forward tunnel.
The tunnel only changes the route Terraform's kubernetes and helm
providers take. It does not close the public endpoint: with the defaults
the cluster still has endpoint_public_access = true, restricted to
eks_public_access_cidrs (or your auto-detected IP). To close the public
endpoint as well, set eks_public_access = false (see
Closing the public endpoint below). The cluster already has
endpoint_private_access = true and an in-VPC SSM-managed EC2 node, which the
tunnel uses as its jump host.
This path keeps TLS validation intact: the providers connect to
127.0.0.1:<port> but set tls_server_name to the real cluster hostname, so
the cluster's CA certificate is still verified. There is no insecure = true
and no 0.0.0.0/0 exposure.
The symptom that calls for this path is the same Kubernetes cluster
unreachable: ... i/o timeout failure partway through terraform apply — but
here it persists even with the correct eks_public_access_cidrs, because the
host has no network route to the public endpoint at all. This reuses the
existing in-VPC node (no bastion to stand up, no extra cost).
- Start the tunnel in one shell and leave it running for the whole apply. The
cluster name, private endpoint, VPC, and SSM instance are all discovered at
runtime — no values are hardcoded (the port defaults to
9443):
scripts/eks-api-tunnel.sh 9443
Requires awscli v2, the session-manager-plugin, and jq. If your
account has more than one EKS cluster or more than one SSM instance in the
cluster VPC, pass the cluster name as the second argument or set
EKS_CLUSTER_NAME / SSM_INSTANCE_ID.
- In another shell, run Terraform pointing the providers at that local port.
Set
eks_api_local_portto the same port, either via the environment:
export TF_VAR_eks_api_local_port=9443
terraform apply
or in tfvars:
# terraform.tfvars
eks_api_local_port = 9443
Before a long apply, confirm the tunnel reaches the cluster. A 401 response
means the API server answered (the request simply carried no token):
curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1:9443/version
Both shells must use the same AWS credentials and Region, for example by
exporting the same AWS_PROFILE and AWS_REGION in each. The credentials must
also stay valid for the whole apply (often 30 minutes or more). Short-lived keys
pasted into a shell can expire partway through, which drops the tunnel. Prefer
a profile that refreshes itself, such as IAM Identity Center (aws sso login)
or a credential_process profile.
The tunnel is needed for every command that talks to the Kubernetes API,
including terraform destroy. Start it before tearing the sample down, or the
destroy fails with the same Kubernetes cluster unreachable error.
Leave eks_api_local_port unset (null, the default) for the normal public
behavior — nothing changes unless you opt in. For a durable production posture,
prefer running Terraform from inside the VPC (in-VPC CI, Cloud9, or a bastion)
so the private endpoint is used natively and the public endpoint can be closed
entirely.
Closing the public endpoint¶
Set eks_public_access = false to disable the EKS public endpoint
(endpoint_public_access = false). eks_public_access_cidrs is then ignored.
Terraform must reach the cluster over the private endpoint for every apply
after that, either from inside the VPC or through the SSM tunnel.
The tunnel needs the cluster and its in-VPC SSM node to exist first, so a fresh deployment from outside the VPC is a two-step apply:
- Create the network and cluster. This step does not use the Kubernetes API, so it works with the public endpoint already closed:
terraform apply \
-var 'eks_public_access=false' \
-target=module.unreal_cloud_ddc_vpc \
-target=module.unreal_cloud_ddc_infra
- Start the tunnel (
scripts/eks-api-tunnel.sh 9443) and run the full apply through it:
terraform apply -var 'eks_public_access=false' -var 'eks_api_local_port=9443'
To close the endpoint on an existing deployment, start the tunnel and run step
2 directly. Keep eks_public_access = false (for example in
terraform.tfvars) on every later apply, or Terraform will reopen the public
endpoint.
Troubleshooting the tunnel¶
| Symptom | Likely cause | Fix |
|---|---|---|
dial tcp 127.0.0.1:9443: connect: connection refused |
The tunnel is not running | Start scripts/eks-api-tunnel.sh 9443 and leave it open |
Connection to destination port failed in the tunnel shell, TLS handshake timeout in Terraform |
The jump host cannot reach the private endpoint on 443 | Set SSM_INSTANCE_ID to one of the cluster's EKS nodes (tagged eks:cluster-name) |
ExpiredToken or the tunnel exits mid-apply |
Credentials expired | Refresh credentials, restart the tunnel, re-run the apply |
Error names the cluster's public *.eks.amazonaws.com URL |
eks_api_local_port is not set |
Set eks_api_local_port to the tunnel port |
no Online SSM-managed instances found |
Nodes are still starting, or the SSM agent cannot reach SSM | Wait for the node groups to be ACTIVE, then retry |
Postdeployment¶
The sample deploys a Route53 dns record that you can use to access your Unreal DDC cluster. This record points to an NLB which may take more time to become fully available when the deployment is complete. You can view the provisioning status of this NLB on the EC2 load balncing screen.
The Unreal Cloud DDC module creates a Service Account and valid bearer token for testing. This bearer token is stored in AWS Secrets Manager. The ARN of this secret is provided as a Terraform output ("unreal_cloud_ddc_bearer_token_arn") on the console following deployment. To fetch the bearer token you can use the aws CLI:
aws secretsmanager get-secret-value --secret-id <"unreal_cloud_ddc_bearer_token_arn">
To validate you can put an object you can run:
curl http://<unreal_ddc_url>/api/v1/refs/ddc/default/00000000000000000000000000000000000000aa -X PUT --data 'test' -H 'content-type: application/octet-stream' -H 'X-Jupiter-IoHash: 4878CA0425C739FA427F7EDA20FE845F6B2E46BA' -i -H 'Authorization: ServiceAccount <secret-manager-token>'
After running this you should get a response that looks as the following:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 29 Jan 2025 19:15:05 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server-Timing: blob.put.FileSystemStore;dur=0.1451;desc="PUT to store: 'FileSystemStore'",blob.put.AmazonS3Store;dur=267.0449;desc="PUT to store: 'AmazonS3Store'",blob.get-metadata.FileSystemStore;dur=0.0406;desc="Blob GET Metadata from: 'FileSystemStore'",ref.finalize;dur=7.1407;desc="Finalizing the ref",ref.put;dur=25.2064;desc="Inserting ref"
{"needs":[]}%
You can then access the same chunk with the following command:
curl http://<unreal_ddc_url>/api/v1/refs/ddc/default/00000000000000000000000000000000000000aa.json -i -H 'Authorization: ServiceAccount <unreal-cloud-ddc-bearer-token>'
The response should look like the following:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 29 Jan 2025 19:16:46 GMT
Content-Type: application/json
Content-Length: 66
Connection: keep-alive
X-Jupiter-IoHash: 7D873DCC262F62FBAA871FE61B2B52D715A1171E
X-Jupiter-LastAccess: 01/29/2025 19:16:46
Server-Timing: ref.get;dur=0.0299;desc="Fetching Ref from DB"
{"RawHash":"4878ca0425c739fa427f7eda20fe845f6b2e46ba","RawSize":4}%
For a more comprehensive test of your deployment, we recommend using the bench marking tools. To do so we used a x2idn.32xlarge as it matched Epic's benchmarking instance to test their configuration.
With the benchmarking tools we ran the following command after compiling the docker image:
docker run --network host jupiter_benchmark --seed --seed-remote --host http://<unreal_ddc_url> --namespace ddc \
--header="Authorization: ServiceAccount <unreal-cloud-ddc-bearer-token>" all
Just a note here, you will have to specify the namespace to be DDC as the token only has access to that namespace.
It is recommended that if you are using this in a production capacity you change the authentication mode from Service Account to Bearer and use an IDP to authenticate and TLS termination.
This sample also deploys a ScyllaDB monitoring stack, enabling real-time insights into the status and performance of your ScyllaDB nodes. The monitoring stack includes Prometheus for metrics collection, Alertmanager for handling alerts, and Grafana for visualization. You can access the Grafana dashboard by using the "monitoring_url" provided in the sample outputs. To learn more about the ScyllaDB monitoring stack, refer to the ScyllaDB Monitoring Stack Documentation.
Requirements¶
| Name | Version |
|---|---|
| terraform | >= 1.10.3 |
| aws | >= 5.89.0 |
| awscc | >= 1.26.0 |
| helm | >= 2.9.0, < 3.0.0 |
| http | >= 3.4.5 |
| kubernetes | >= 2.24.0 |
| random | 3.7.2 |
Providers¶
| Name | Version |
|---|---|
| aws | >= 5.89.0 |
| awscc | >= 1.26.0 |
| http | >= 3.4.5 |
Modules¶
| Name | Source | Version |
|---|---|---|
| unreal_cloud_ddc_infra | ../../modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-infra | n/a |
| unreal_cloud_ddc_intra_cluster | ../../modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-intra-cluster | n/a |
| unreal_cloud_ddc_vpc | ./vpc | n/a |
Resources¶
Inputs¶
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| github_credential_arn | Github Credential ARN | string |
n/a | yes |
| route53_public_hosted_zone_name | The root domain name for the Hosted Zone where the ScyllaDB monitoring record should be created. | string |
n/a | yes |
| allow_my_ip | Automatically add your IP to the security groups allowing access to the Unreal DDC and SycllaDB Monitoring load balancers | bool |
true |
no |
Outputs¶
| Name | Description |
|---|---|
| monitoring_url | n/a |
| unreal_cloud_ddc_bearer_token_arn | n/a |
| unreal_ddc_url | n/a |
Requirements¶
| Name | Version |
|---|---|
| terraform | >= 1.10.3 |
| aws | >= 5.89.0 |
| awscc | >= 1.26.0 |
| helm | >= 2.9.0 |
| http | >= 3.4.5 |
| kubernetes | >= 2.24.0 |
| random | 3.7.2 |
Providers¶
| Name | Version |
|---|---|
| aws | >= 5.89.0 |
| awscc | >= 1.26.0 |
| http | >= 3.4.5 |
Modules¶
| Name | Source | Version |
|---|---|---|
| unreal_cloud_ddc_infra | ../../modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-infra | n/a |
| unreal_cloud_ddc_intra_cluster | ../../modules/unreal/unreal-cloud-ddc/unreal-cloud-ddc-intra-cluster | n/a |
| unreal_cloud_ddc_vpc | ./vpc | n/a |
Resources¶
Inputs¶
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| allow_my_ip | Automatically add your IP to the security groups allowing access to the Unreal DDC and SycllaDB Monitoring load balancers | bool |
true |
no |
| github_credential_arn | Github Credential ARN | string |
n/a | yes |
| route53_public_hosted_zone_name | The root domain name for the Hosted Zone where the ScyllaDB monitoring record should be created. | string |
n/a | yes |
Outputs¶
| Name | Description |
|---|---|
| monitoring_url | n/a |
| unreal_cloud_ddc_bearer_token_arn | n/a |
| unreal_ddc_url | n/a |