Control Plane Operations Platform

Multi-account GitOps platform for managing EKS clusters and workloads declaratively using KRO + ACK + ArgoCD capabilities on EKS, with AI-assisted operations via ArgoCD MCP on AWS Bedrock AgentCore.

Documentation

• Architecture Overview — multi-account topology, account roles, IAM model
• Repository Ecosystem & CI/CD Pipeline — CodeCommit repos, pipeline stages, end-to-end flow
• ArgoCD ApplicationSets — ApplicationSet types, discovery patterns, platform claim processing
• Modules — bootstrap, agentcore-mcp, catalog, and runtime reference

Repository Structure

control-plane-operations/
├── bootstrap/                              # Terraform — Control Plane Account
│   ├── main.tf                             # VPC, EKS (Auto Mode), CodeCommit, seed repo
│   ├── iam.tf                              # control-plane-admin role + Pod Identity
│   ├── capabilities.tf                     # ACK, KRO, ArgoCD EKS capabilities
│   ├── credit-card-pipeline.tf             # CI/CD: CodePipeline + CodeBuild
│   ├── credit-card-pipeline-seed.tf        # Seeds Java app + Helm chart repos
│   ├── templates/                          # ArgoCD manifest templates (.tftpl)
│   ├── variables.tf
│   ├── outputs.tf
│   └── versions.tf
│
├── agentcore-mcp/                          # Terraform — GenAI MCP Account
│   ├── main.tf                             # ECR, Secrets Manager, CodeBuild
│   ├── runtime.tf                          # IAM, AgentCore Runtime
│   ├── cognito.tf                          # Cognito User Pool, OAuth (M2M)
│   ├── docker/argocd-mcp/
│   │   ├── Dockerfile                      # ARM64 image with AWS CLI
│   │   └── entrypoint.sh                   # Fetches API token from Secrets Manager
│   ├── variables.tf
│   ├── outputs.tf
│   └── versions.tf
│
├── catalog/                                # KRO Resource Graph Definitions
│   ├── eks-cluster.yaml                    # RGD: EKS cluster (Auto Mode via ACK)
│   ├── helm-install.yaml                   # RGD: ArgoCD App for Helm charts
│   └── spoke-account-iam.yaml             # RGD: IAMRoleSelector per spoke
│
├── runtime/                                # Runtime claims (per environment)
│   ├── development/
│   │   └── credit-card/
│   │       ├── data-plane/claim.yaml       # EKS cluster + workloads claim
│   │       └── apps/                       # App configs for ApplicationSets
│   ├── staging/
│   │   └── credit-card/
│   └── production/
│       └── credit-card/
│
├── applications/                           # Helm chart overrides per environment
│   ├── development/
│   ├── staging/
│   └── production/
│
└── spoke-account-preparation/              # IAM role setup for spoke accounts
    ├── control-plane-execution-policy.json
    ├── control-plane-execution-trust.json
    └── README.md

Deployment Order

0. Create Terraform Remote State Bucket

Before running any Terraform module, create the S3 bucket for remote state.

Open create-tfstate-bucket.sh and fill in the variables at the top:

PROFILE=""          # Your AWS CLI profile name, or leave empty for default credentials
REGION="us-east-1"  # AWS region where the bucket will be created
ACCOUNT_ID=""       # Your AWS account ID (e.g. 123456789012)

Then run:

./create-tfstate-bucket.sh

This creates a versioned, encrypted, private S3 bucket named psp-tfstate-<ACCOUNT_ID>.

Important: Before running terraform init in any module, update the backend "s3" block in its versions.tf with your bucket name and (optionally) your AWS profile:

> backend "s3" {

1. Spoke Account Preparation

Before creating the role, update the trust policy with your hub account ID:

# Replace REPLACEWITHHUBACCOUNTID with your actual hub account ID
sed -i '' 's/REPLACEWITHHUBACCOUNTID/123456789012/g' \
  spoke-account-preparation/control-plane-execution-trust.json

Then in each spoke account, create the execution role:

aws iam create-role --role-name control-plane-execution \
  --assume-role-policy-document file://spoke-account-preparation/control-plane-execution-trust.json
aws iam put-role-policy --role-name control-plane-execution \
  --policy-name control-plane-execution-policy \
  --policy-document file://spoke-account-preparation/control-plane-execution-policy.json

Workshop note: If you cloned this repo for a workshop, the trust policy may already contain the hub account ID. Verify the value in spoke-account-preparation/control-plane-execution-trust.json before proceeding. For single-account deployments, the hub and spoke account IDs are the same.

Allow ECR Cross-Account Replication

If enableecrreplication = true (multi-account deployments), the spoke account's ECR registry must allow the hub account to replicate images into it. Without this, replication fails with DESTINATIONREGISTRYACCESS_DENIED.

aws ecr put-registry-policy \
  --policy-text '{
    "Version": "2012-10-17",
    "Statement": [{
      "Sid": "AllowReplicationFromHub",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::<HUBACCOUNTID>:root"},
      "Action": ["ecr:ReplicateImage", "ecr:BatchImportUpstreamImage", "ecr:CreateRepository"],
      "Resource": "arn:aws:ecr:us-east-1:<SPOKEACCOUNTID>:repository/*"
    }]
  }' \
  --region us-east-1 \
  --profile <your-spoke-profile>

Replace <HUBACCOUNTID> with the account ID where the CI/CD pipeline runs, and <SPOKEACCOUNTID> with the destination spoke account ID.

Tag Spoke Subnets for EKS Auto Mode

EKS Auto Mode uses subnet tags to discover where to launch nodes. Without these tags, the node provisioner (Karpenter) cannot find subnets and instances will never be created.

Tag the spoke subnets that will be used by the EKS cluster:

# For public subnets (MapPublicIpOnLaunch=true)
aws ec2 create-tags \
  --resources subnet-0aaa111bbb222ccc3 subnet-0ddd444eee555fff6 subnet-0ggg777hhh888iii9 \
  --tags Key=kubernetes.io/role/elb,Value=1 \
  --region us-east-1 \
  --profile <your-spoke-profile>
For private subnets (recommended for production)
aws ec2 create-tags \
  --resources subnet-0aaa111bbb222ccc3 subnet-0ddd444eee555fff6 subnet-0ggg777hhh888iii9 \
  --tags Key=kubernetes.io/role/internal-elb,Value=1 \
  --region us-east-1 \
  --profile <your-spoke-profile>

Important: Use the same subnet IDs you will set in spokesubnetids in your .terraform.tfvars. Without this step, the EKS cluster will be created but nodes will remain in a pending state indefinitely.

Enable GuardDuty EKS Runtime Monitoring

If your claim includes the aws-guardduty-agent addon, you must enable both RUNTIMEMONITORING and EKSRUNTIME_MONITORING on the GuardDuty detector in the spoke account. Without this, the agent will crash with AccessDeniedException even if the IAM role and Pod Identity are correctly configured.

# Get the detector ID
DETECTOR_ID=$(aws guardduty list-detectors --region us-east-1 --profile <your-spoke-profile> --query 'DetectorIds[0]' --output text)
Enable EKS Runtime Monitoring (set EKSADDONMANAGEMENT=DISABLED since the addon is managed by the platform)
aws guardduty update-detector \
  --detector-id $DETECTOR_ID \
  --features '[{"Name":"EKSRUNTIMEMONITORING","Status":"ENABLED","AdditionalConfiguration":[{"Name":"EKSADDONMANAGEMENT","Status":"DISABLED"}]},{"Name":"RUNTIMEMONITORING","Status":"ENABLED","AdditionalConfiguration":[{"Name":"EKSADDONMANAGEMENT","Status":"DISABLED"},{"Name":"ECSFARGATEAGENTMANAGEMENT","Status":"DISABLED"},{"Name":"EC2AGENTMANAGEMENT","Status":"DISABLED"}]}]' \
  --region us-east-1 \
  --profile <your-spoke-profile>

Note: EKSADDONMANAGEMENT is set to DISABLED because the platform manages the addon via the EKS cluster catalog RGD. Enabling it would cause GuardDuty to conflict with the platform-managed addon.

2. Control Plane (Hub Account)

Configure Variables

Tip: Run ./gather-infra.sh to discover your VPC IDs, subnet IDs, Organization ID, IAM Identity Center instance ARN, and group IDs. Before running, open the file and fill in the variables at the top (PROFILE, REGION, ACCOUNT_ID). Use the output to fill in your .terraform.tfvars.

Create bootstrap/.terraform.tfvars with your environment values:

# AWS Region
region = "us-east-1"
AWS CLI Profile (leave empty or remove for default credentials)
aws_profile = ""
Set to false for single-account deployments
enableecrreplication = false
EKS Cluster
cluster_name       = "control-plane"
kubernetes_version = "1.35"
VPC
vpc_cidr = "10.0.0.0/16"
AWS Organizations
organization_id = "o-abc123def4"
CodeCommit repository name (ArgoCD GitOps source)
gitrepourl = "control-plane-operations"
IAM Identity Center (SSO) — used by ArgoCD capability
idcinstancearn = "arn:aws:sso:::instance/ssoins-1234567890abcdef"
ArgoCD RBAC — IAM Identity Center group IDs
argocdadmingroup_id    = "a1b2c3d4-5678-90ab-cdef-111111111111"
argocdreadonlygroup_id = "a1b2c3d4-5678-90ab-cdef-222222222222"
Spoke account (development environment)
spokeaccountid         = "123456789012"
spoke_region             = "us-east-1"
spokevpcid             = "vpc-0abc123def456789a"
spokesubnetids         = ["subnet-0aaa111bbb222ccc3", "subnet-0ddd444eee555fff6"]
spokekubernetesversion = "1.33"
Tags
tags = {
  ManagedBy   = "terraform"
  Project     = "control-plane"
  Environment = "hub"
}

| Variable | Required | Description | |---|---|---| | region | No (default: us-east-1) | AWS region for the control plane | | aws_profile | No (default: "") | AWS CLI profile name for authentication. Required when using named profiles instead of default credentials | | enableecrreplication | No (default: true) | Enable ECR cross-account replication. Set to false for single-account deployments | | cluster_name | No (default: control-plane) | Name of the hub EKS cluster | | kubernetes_version | No (default: 1.35) | Kubernetes version for the hub cluster | | vpc_cidr | No (default: 10.0.0.0/16) | CIDR block for the hub VPC | | organization_id | Yes | AWS Organizations ID (e.g. o-abc123def4) — used in cross-account trust policies | | gitrepourl | No (default: control-plane-operations) | CodeCommit repository name | | idcinstancearn | Yes | ARN of the IAM Identity Center instance — enables SSO login to ArgoCD UI | | argocdadmingroup_id | Yes | IAM Identity Center group ID for ArgoCD ADMIN role | | argocdreadonlygroup_id | Yes | IAM Identity Center group ID for ArgoCD VIEWER role | | spokeaccountid | Yes | AWS account ID of the spoke (data-plane) account | | spoke_region | No (default: us-east-1) | Region for the spoke EKS cluster | | spokevpcid | Yes | VPC ID in the spoke account | | spokesubnetids | Yes | Subnet IDs in the spoke account for the EKS cluster | | spokekubernetesversion | No (default: 1.33) | Kubernetes version for spoke clusters | | tags | No | Common tags applied to all resources |

Apply

cd bootstrap
terraform init
terraform apply -var-file=.terraform.tfvars

Adding a New Runtime

• Create runtime/<environment>/<name>/data-plane/claim.yaml
• Include SpokeAccountIAM, EKSCluster, and HelmInstall instances
• Set namespace to <name>-<environment>
• Push to Git — ArgoCD picks it up automatically via the ApplicationSet

Deploying an Application

Bootstrap seeds two CodeCommit repositories and a pipeline (credit-card-pipeline) as a ready-to-use example. The same pattern applies to any new application.

Repositories created by bootstrap

| Repository | Purpose | |---|---| | credit-card-java | Java application source — Dockerfile, buildspec | | credit-card-data-plane | Helm chart — Chart.yaml, values.yaml, templates/ |

Trigger a deployment

Push to the main branch of the app repo to start the pipeline:

# Clone the seeded app repo
git clone codecommit::us-east-1://credit-card-java
cd credit-card-java
Make changes, then push
git add .
git commit -m "update application"
git push origin main

The pipeline runs three stages automatically:

• Build — compiles the Java app, builds and pushes the Docker image to ECR
• Package — packages and pushes the Helm chart as an OCI artifact to ECR
• Update — commits updated config.json and values.yaml to this platform repo

Monitor the pipeline

# List pipeline executions
aws codepipeline list-pipeline-executions --pipeline-name credit-card-pipeline
Watch ArgoCD sync status
kubectl get applications -n argocd

Update Helm values

To change resource limits, replicas, or any other Helm value without a code change, edit the relevant file and push:

runtime/development/credit-card/apps/values.yaml

ArgoCD picks up the change and re-syncs without running the pipeline.

Optional: AI-Assisted Operations (ArgoCD MCP + Bedrock AgentCore)

Deploys an ArgoCD MCP server on Bedrock AgentCore so AI agents can query and operate ArgoCD via natural language.

Deploy ArgoCD MCP (GenAI Account)

Tip: If running in a single-account setup, the ArgoCD server URL comes from your bootstrap outputs: cd ../bootstrap && terraform output -raw argocdserverurl

Create agentcore-mcp/.terraform.tfvars:

region            = "us-east-1"
argocdserverurl = "https://abc123def456.eks-capabilities.us-east-1.amazonaws.com"
argocdapitoken  = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
tags = {   ManagedBy = "terraform"   Project   = "argocd-mcp" }

| Variable | Required | Description | |---|---|---| | region | No (default: us-east-1) | AWS region for the AgentCore deployment | | argocdserverurl | Yes | ArgoCD server URL — terraform output -raw argocdserverurl from the bootstrap folder | | argocdapitoken | Yes (sensitive) | Project-scoped JWT from ArgoCD UI: Settings → Projects → default → Roles → JWT Tokens → Generate | | tags | No | Common tags applied to all resources |

The argocdapitoken is sensitive — pass via CLI to avoid storing it in the tfvars file:

> terraform apply -var='argocdapitoken=eyJhbGci...'

cd agentcore-mcp
terraform init
terraform apply -var-file=.terraform.tfvars

Register MCP in DevOps Agent

cd agentcore-mcp
terraform output devopsagentclient_id
terraform output -raw devopsagentclient_secret
terraform output devopsagentexchange_url
terraform output devopsagentoauth_scopes
terraform output devopsagentmcpendpointurl

Validate the MCP Endpoint

Open test-mcp.sh and set the region at the top if needed:

REGION="us-east-1"  # AWS region where AgentCore is deployed

Then run:

./test-mcp.sh

Verifies the full OAuth → MCP → ArgoCD chain: obtains a Cognito token, performs the MCP handshake, lists tools, and calls list_applications.

Token Management

To rotate the ArgoCD API token:

cd agentcore-mcp terraform apply -var='argocdapitoken=<new-token>'

Rebuilding the Docker Image

If you modify the Dockerfile or entrypoint:

cd agentcore-mcp terraform taint nullresource.argocdmcpbuildtrigger terraform apply

Note: The Dockerfile uses ECR Public mirrors (public.ecr.aws/docker/library/) instead of Docker Hub to avoid rate limits. If building in environments without ECR Public access, swap back to node:20-slim in the Dockerfile.