Lights Off!

Request that Service Quotas → AWS services → AWS Lambda → Concurrent executions be increased. The default is 1000 .

Lights Off needs 1 unit for a time-critical function. New AWS accounts start with a quota of 10, but Lambda always holds back 10, which leaves 0 available! Within a given AWS account, the quota is set separately for each region.

RDS and Aurora automatically start stopped databases after 7 days. Install github.com/sqlxpert/step-stay-stopped-aws-rds-aurora to re-stop them automatically, or set a once-a-week sched-start and add days to sched-stop :

|Locations|sched-start|sched-stop| |:---|:---:|:---:| |USA Mainland|uTH:M=6T02:30|d=_ H:M=03:30| |North America (from Hawaii to Newfoundland)|uTH:M=6T04:30|d=_ H:M=05:30| |Europe|uTH:M=5T18:30|d=_ H:M=19:30| |India|uTH:M=5T13:00|d=_ H:M=14:00| |North America + Europe|uTH:M=6T04:30|⚠ u=6 u=7 H:M=05:30| |North America + Europe + India|uTH:M=6T04:30|⚠ u=6 u=7 H:M=05:30| |Europe + India|uTH:M=5T18:30|d=_ H:M=19:30|

• If the database usually takes longer than 1 hour to start, change

• For most time zone combinations, sched-stop can be made daily. If a

• ⚠ For North America + Europe [+ India], stop

• Set the database's weekly maintenance window to a time period when the

The sched-start tag works for EC2 instances with EBS volumes if:

• Your EBS volumes are unencrypted, or
• You use the default, AWS-managed aws/ebs encryption key, or
• You use custom keys in the same AWS account as each EC2 instance, the key

Because your custom keys are in a different AWS account than your EC2 instances, you must add a statement like the following to the key policies:

{
      "Sid": "LightsOffEc2StartInstancesWithEncryptedEbsVolumes",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "kms:CreateGrant",
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:PrincipalOrgPaths": "o-ORGID/r-ROOTID/ou-PARENTORGUNIT_ID/*"
        },
        "ArnLike": {
          "aws:PrincipalArn": "arn:aws:iam::ACCOUNT:role/LightsOff-DoLambdaFnRole-*"
        },
        "StringLike": {
          "kms:ViaService": "ec2.*.amazonaws.com"
        },
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }

• One account: Delete the entire "ForAnyValue:StringLike" section and

• AWS Organizations: Replace ACCOUNT with * and o-ORGID , r-ROOTID ,

>If an EC2 instance does not start as scheduled, a KMS key permissions error is possible.

Because you want to use the sched-backup tag in a complex AWS environment, you must address the following AWS Backup requirements:

1. Vault

AWS Backup creates the Default vault the first time you open the list of vaults in a given AWS account and region, using the AWS Console. Otherwise, see Backup vault creation and AWS::Backup::BackupVault or awsbackupvault . Update the BackupVaultName CloudFormation stack parameter if necessary.

2. Vault policy

If you have added "Deny" statements, be sure that DoLambdaFnRole still has access.

3. Backup role

AWS Backup creates AWSBackupDefaultServiceRole the first time you make a backup in a given AWS account using the AWS Console (AWS Backup → My account → Dashboard → On-demand backup). Otherwise, see Default service role for AWS Backup. Update the BackupRoleName parameter if necessary.

4. KMS key policies

AWSBackupDefaultServiceRole works if:

- Your EBS volumes and RDS/Aurora databases are unencrypted, or - You use the default, AWS-managed aws/ebs and aws/rds encryption keys, or - You use custom keys in the same AWS account as each disk and database, the key policies contain the default "Enable IAM User Permissions" statement, and they do not contain "Deny" statements.

If your custom keys are in a different AWS account than your disks and databases, you must modify the key policies. See Encryption for backups in AWS Backup, How EBS uses KMS, Overview of encrypting RDS resources, and Key policies in KMS.

>If no backup jobs appear in AWS Backup, or if jobs do not start, a permissions problem is likely.

The state of an AWS resource might change between the "Find" and "Do" steps; this sequence is fundamentally non-atomic. An operation might also be repeated due to queue message delivery logic; operations are idempotent. If a state change is favorable or an operation is repeated, Lights Off logs success responses or expected exceptions (depending on the AWS service) at the INFO level. For RDS database instance start/stop operations, however, expected exceptions are logged at the ERROR level because Lights Off cannot determine whether they represent actual errors or harmless repetition (such as trying to start a database instance that has already been started).

For complete details, see the technical article Idempotence: Doing It More than Once.

Two strengths of this tool are its distributed design and its simplicity.

Lights Off operates independently in each region, in each AWS account. Every (region, account) pair has its own log and error queue. Operation does not depend on central resources, other than an optional customer-managed multi-region KMS key. Centralized logging and monitoring would introduce a single point of failure and add complexity, only to duplicate AWS features that can cover all of your applications.

Consider monitoring Lights Off through...

• The organization CloudTrail

• CloudWatch Logs data centralization

The fetish for metrics, dashboards, and alerts raises a deeper question. Is it worth paging someone over an unsuccessful EC2 instance stop request? Probably not! If large amounts of money are at stake, you need AWS Budgets. The thresholds, notifications, and actions you define there cover any and all tools, applications, provisioning processes, and people, not just Lights Off.

Backups are worth monitoring, but at the point of consumption, not at the point of creation. No matter how you create backups, you should implement an automated process to restore critical ones (the first hourly backup of the day, for example) and validate the results. AWS Backup restore testing solves this problem.

When your organization becomes large and formal, you will graduate from friendly, locally-managed sched-backup tags to centralized backup plans. Then, you can use AWS Backup Audit Manager to continuously check your resources, your plans, and the presence of your backups. (You will still have to validate the backups.) Consider quitting before your job shifts from making software to making PowerPoint presentations.

Employees, consultants, and vendors who demonstrate pretty dashboards should instead start by asking what is material to you. You should ask whether AWS provides standard ways to gather the information you actually care about.

---

The lightsoffstacksetoperationpreferences module variable lets you customize StackSet operation concurrency. Preferences apply to the StackSet as a whole, if you update the CloudFormation template or change a parameter. They also apply to StackSet instances affected when you change the region list or the organizational unit list.

For example, to allow up to 4 parallel deployments and accept up to 3 failures before cancellation (the module's conservative defaults are 2 and 2), add:

# module "lightsoffstackset" {
        # ...other attributes...
lightsoffstacksetoperationpreferences = {           maxconcurrentcount    = 4           failuretolerancecount = 3         }

**⚠ Do not change lightsoffstacksetoperationpreferences after module creation**, until github.com/hashicorp/terraform-provider-aws/issues/39393 has been resolved. This bug was still present as of 2026-07-01, in Terraform AWS Provider v6.53.0 . Work-arounds:

- Empty lightsoffstacksetorganizationalunit_ids to delete all existing StackSet instances. Then, restore the list values and change lightsoffstacksetoperationpreferences to create StackSet instances using the new preferences, or - Employ a blue/green deployment strategy by creating a second module instance with a different or non-empty lightsoffstacksetnamesuffix .

---

---

Your CloudFormation StackSet deployment targets might be more complex than the cross product lightsoffstacksetregions × lightsoffstacksetorganizationalunitids . You might want to deploy Lights Off in different regions, depending on the organizational unit, or in different OUs, depending on the region. You might also want to include or exclude specific AWS account numbers. You can also selectively override CloudFormation parameter values.

|Goal|Documentation| |:---|:---| |Vary regions, organizational unit, and account combinations|DeploymentTargets
awscloudformationstacksetinstance.deploymenttargets| |Override parameters|awscloudformationstacksetinstance.parameteroverrides|

To define all Lights Off StackSet instances yourself, leave the lightsoffstacksetorganizationalunit_ids list empty, or do not set this module variable at all.

To define some or all StackSet instances yourself, add one or more resource blocks after the module block:

locals {
        operationpreferences = module.lightsoffstackset.lightsoffstacksetoperation_preferences
      }
resource "awscloudformationstacksetinstance" "lightsoffcustom" {         stacksetname = module.lightsoffstackset.lightsoffstackset_name
stacksetinstance_region = "us-east-1"         # Specify an element of local.operationpreferences["regionorder"]         # or use         # foreach = toset(local.operationpreferences["region_order"])
operation_preferences {           concurrencymode             = local.operationpreferences["concurrency_mode"]           regionconcurrencytype      = local.operationpreferences["regionconcurrency_type"]           regionorder                 = local.operationpreferences["region_order"]           maxconcurrentpercentage    = local.operationpreferences["maxconcurrent_percentage"]           maxconcurrentcount         = local.operationpreferences["maxconcurrent_count"]           failuretolerancepercentage = local.operationpreferences["failuretolerance_percentage"]           failuretolerancecount      = local.operationpreferences["failuretolerance_count"]         }
# ...other attributes...       }

Arbitrary HashiCorp Configuration Language (HCL) syntax restrictions forbid assigning an entire object value to a block, and instead require assigning the attribute values one by one. See also "Attributes as Blocks" in HashiCorp's Terraform Language Documentation.

---

CloudFormation Stack Least-Privilege

You can use a CloudFormation service role to delegate only the privileges needed to create the LightsOff stack. (This is done for you if you use Terraform at Step 3 of the Quick Start.)

First, create the LightsOffPrereq stack from cloudformation/lightsoffaws_prereq.yaml .

Under "Additional settings" → "Stack policy - optional", you can "Upload a file" and select a locally-saved copy of cloudformation/lightsoffawsprereq_policy.json . The stack policy prevents inadvertent replacement or deletion of the deployment role during stack updates, but it cannot prevent deletion of the entire LightsOffPrereq stack.

Next, when you create the LightsOff stack from cloudformation/lightsoff_aws.yaml , set "Permissions - optional" → "IAM role - optional" to LightsOffPrereq-DeploymentRole . If your own privileges are limited, you might need permission to pass the deployment role to CloudFormation. See the LightsOffPrereq-SampleDeploymentRolePassRolePol IAM policy for an example.

CloudFormation StackSet Least-Privilege

For a CloudFormation StackSet, you can use self-managed permissions by copying the inline IAM policy of LightsOffPrereq-DeploymentRole to a customer-managed IAM policy, attaching your policy to AWSCloudFormationStackSetExecutionRole and propagating the policy and the role policy attachment to all target AWS accounts.

Terraform Least-Privilege

If you do not give Terraform full AWS administrative permissions, you must give it permission to:

• List, describe, get tags for, create, tag, update, untag and delete

• Create, tag, describe, update, untag and delete

• List, describe, create, tag, update, untag, and delete CloudFormation

• Set and get CloudFormation stack policies
• Pass LightsOffPrereq-DeploymentRole-* to CloudFormation
• List, describe, and get tags for, all data sources. For a list, run:

grep 'data "' terraform/.tf | cut --delimiter=' ' --fields='1,2' | sort | uniq

Open the AWS Service Authorization Reference, go through the list of services on the left, and consult the "Actions" table for each of:

• AWS Identity and Access Management (IAM)
• Amazon S3
• CloudFormation
• AWS Security Token Service
• AWS Backup (if you use the sched-backup tag)
• AWS Key Management Service (if you encrypt the SQS queues and/or the

• AWS Organizations (if you create a CloudFormation StackSet with the

In most cases, you can scope Terraform's permissions to one workload by regulating resource naming and tagging, and then by using:

• ARN patterns in Resource lists
• ARN patterns in Condition entries
• Request tag and then resource tag Condition entries

The basic //terraform module creates the LightsOffPrereq stack, which defines the IAM role that gives CloudFormation the permissions it needs to create the LightsOff stack. Terraform itself does not need the deployment role's permissions.

• Least-privilege roles for the AWS Lambda functions that find resources and

• A least-privilege queue policy. The operation queue can only consume

• Readable IAM policies, broken down into discrete statements by service,

• Optional encryption at rest with the AWS Key Management System (KMS), for

• No data storage other than in queues and logs, with short or configurable

• Tolerance for clock drift in a distributed system. The "Find" function

• An optional CloudFormation service role for least-privilege deployment.

• Only allow trusted people and services to tag AWS resources. A sample

• Prevent people who can set the sched-backup tag from deleting backups.
• Prevent people from modifying components, most of which can be identified by

• Prevent people from directly invoking the AWS Lambda functions and from

• Log infrastructure changes using AWS CloudTrail, and set up alerts.
• Automatically copy backups to an AWS Backup vault in an isolated account.

• Separate production workloads. You might choose not to deploy Lights Off to

• If you use Terraform, do not use it with an AWS access key and do not give it

A sample service control policy is available to prevent tampering with Lights Off schedule tags.

This SCP offers two-way protection: roles subject to the SCP can neither remove nor add schedule tags. They cannot change existing schedule tag values, either.

In your AWS Organizations management account, in the region where you manage infrastructure-as-code templates for non-regional resources, create a CloudFormation stack from cloudformation/scpprotectlightsoff_tags.yaml .

Or, reference the equivalent Terraform module:

module "lightsoffscp" {
  source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform-scp?ref=v3.6.1"
  # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
  # Check that the release is immutable!
scptargetids = [     "ou-0123-abcdefg",   ] }

In either case, specify the number of the account or the ou- ID of the organizational unit that you use for testing SCPs.

Test the SCP before applying it broadly, because it generally reduces existing EC2, EBS, and RDS/Aurora tagging permissions. Human users or automated processes might rely on those permissions. This is especially true of backup restoration, blue/green deployment, and cluster scaling workflows, which might copy tags to new resources.

You will need at least one SCP-exempt role in every AWS account, to manage schedule tags. I recommend IAM Identity Center permission sets. You can customize ScpPrincipalCondition / scpprincipalcondition to reference permission set roles.

The SCP works by denying certain tag addition/change and removal requests. It cannot add permissions that have been denied by another SCP, or that were never allowed by a role's attached or inline policies.

SCPs do not affect roles or other IAM principals in the AWS Organizations management account.

Lights Off can delete and recreate many types of expensive AWS infrastructure in your own CloudFormation stacks, based on cron schedules in stack tags.

Deleting AWS Client VPN resources overnight, while developers are asleep, is a sample use case. See 10-minute AWS Client VPN.

To make your own CloudFormation template compatible, see cloudformation/lightsoffawsbonuscloudformation_example.yaml .

Not every resource needs to be deleted and recreated; condition the creation of expensive resources on the Enable parameter. In the AWS Client VPN stack, the VPN endpoints and VPC security groups are not deleted, because they do not cost anything. The VPN attachments can be deleted and recreated with no need to reconfigure VPN clients.

Set the sched-set-Enable-true and sched-set-Enable-false tags on your own CloudFormation stack and make sure that the EnableSchedCloudFormationOps parameter is set to true (the default) in your Lights Off CloudFormation stack/StackSet or Terraform module. At the scheduled times, Lights Off will perform a stack update, toggling the value of the Enable parameter to true or false. (Capitalize Enable in the tag keys, to match the parameter name.)

If your tagged stack lacks a CloudFormation service role, Lights Off logs an error of "type" STACKNEEDSSERVICE_ROLE in a "Find" log stream in the log. To make scheduled updates possible, you must first perform a stack update in which you attach an IAM role that gives CloudFormation the permissions it needs to manage the resources defined in your stack. See the RoleARN request parameter in the UpdateStack reference.

💡 Because you can attach a different service role by performing a stack update, you may wish to maintain two roles, a less restrictive one that you can use for stack creation, arbitrary modification, and deletion, and a restrictive one that you can leave in place for automated stack updates. The latter role need only support the AWS actions invoked when the stack's Enable parameter changes from true to false and vice versa.

If the status of your tagged stack is other than CREATE_COMPLETE or UPDATE_COMPLETE at the scheduled time, Lights Off logs an error of "type" STACKSTATUSIRREGULAR in a "Find" log stream, instead of attempting an update that is likely to fail and require a rollback. To resume scheduled stack updates, resolve the underlying template error or permissions error and successfully complete one manual stack update.

The sample service control policy does not cover sched-set-Enable-true and sched-set-Enable-false tags on CloudFormation stacks (or StackSets, whose tags would be copied to member stack instances). Because UpdateStack overwrites the entire set of tags, distinguishing between adding a tag, preserving an existing tag's value, explicitly removing a tag, and removing a tag by removing all tags, requires multiple IAM policy statements for each tag key. Only privileged roles should be allowed to create and update CloudFormation stacks/StackSets, including their tags.

Lights Off takes advantage of patterns in boto3, the AWS software development kit (SDK) for Python, and in the underlying AWS API. Adding AWS services, resource types, and operations is easy. For example, supporting Aurora database clusters (RDS database instances were already supported) required adding:

AWSRsrcType(
      "rds",
      ("DB", "Cluster"),
      {
        ("start", ): {},
        ("stop", ): {},
        ("backup", ): {},
      },
      rsrcidkey_suffix="Identifier",
      tags_key="TagList",
    )

Given the words DB and Cluster in the resource type name, plus the operation verb start, the sched-start tag key and the startdbcluster method name are derived mechanically.

If an operation method takes more than just the resource identifier, add a dictionary of static keyword arguments. For complex arguments, sub-class the AWSOp class and override op_kwargs .

The startbackupjob method takes an Amazon Resource Name (ARN), whose format is consistent for all resource types. As long as AWS Backup supports the resource type, there is no extra work to do.

Add statements like the one below to the Identity and Access Management (IAM) policy for the role used by the "Do" AWS Lambda function, to authorize operations. You must of course authorize the role used by the "Find" function to describe (list) resources.

- Effect: Allow
            Action: rds:StartDBCluster
            Resource: !Sub "arn:${AWS::Partition}:rds:${AWS::Region}:${AWS::AccountId}:cluster:*"
            Condition:
              "Null": { "aws:ResourceTag/sched-start": "false" }

Let me know what resource types you'd like me to add!