Sentient Enclaves Framework

A Comprehensive Trusted Execution Environment (TEE) Framework for Secure AI and Cryptographic Applications

Overview • Architecture • Quick Start • Components • Build System • CLI Reference • API Reference • Configuration • Security • Troubleshooting

---

Overview

The Sentient Enclaves Framework is a production-grade, security-focused framework designed for building, deploying, and managing applications within AWS Nitro Enclaves. It provides comprehensive tooling for secure computation in Trusted Execution Environments (TEEs), enabling:

• Confidential AI Inference: Run AI models in isolated, attestable environments
• Secure Cryptographic Operations: Perform sensitive operations with hardware-backed isolation
• Remote Attestation: Cryptographically prove the integrity of enclave workloads
• Secure Communication: Encrypted channels between host and enclave via VSOCK

Key Features

| Feature | Description | |---------|-------------| | 🔐 Hardware Isolation | Leverages AWS Nitro Enclaves for hardware-level memory isolation | | 📝 Remote Attestation | Cryptographic proof of enclave state with VRF proofs and NSM integration | | 🔄 Secure Communication | VSOCK-based Pipeline protocol for host-enclave communication | | 🌐 Network Proxying | Bidirectional port forwarding between host network and enclave | | 📦 Reproducible Builds | Deterministic EIF image generation with consistent PCR values | | 🛠️ Custom Init System | Lightweight service manager with dependency resolution | | 📊 File System Monitoring | Real-time integrity monitoring with NATS integration |

---

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           HOST EC2 INSTANCE                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐          │
│  │  Pipeline CLI   │  │  PF-Proxy (FW)  │  │  PF-Proxy (RV)  │          │
│  │    (Client)     │  │   Host-side     │  │   Host-side     │          │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘          │
│           │ VSOCK              │ VSOCK              │ VSOCK             │
├───────────┼────────────────────┼────────────────────┼───────────────────┤
│           │                    │                    │                   │
│  ┌────────▼────────────────────▼────────────────────▼────────────────┐  │
│  │                     AWS NITRO ENCLAVE                             │  │
│  │  ┌─────────────────────────────────────────────────────────────┐  │  │
│  │  │                    enclave-init (PID 1)                     │  │  │
│  │  │         Service Manager with Dependency Resolution          │  │  │
│  │  └─────────────────────────────────────────────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐  │  │
│  │  │   Pipeline    │  │  ra-web-srv   │  │     fs-monitor        │  │  │
│  │  │   (Server)    │  │  (HTTPS API)  │  │  (inotify+NATS)       │  │  │
│  │  └───────────────┘  └───────────────┘  └───────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐  │  │
│  │  │  NATS Server  │  │ PF-Proxy(FW)  │  │    PF-Proxy(RV)       │  │  │
│  │  │ (Event Bus)   │  │  Guest-side   │  │    Guest-side         │  │  │
│  │  └───────────────┘  └───────────────┘  └───────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌─────────────────────────────────────────────────────────────┐  │  │
│  │  │           NSM (Nitro Security Module) Interface             │  │  │
│  │  │    Attestation Documents • RNG • PCR Registers              │  │  │
│  │  └─────────────────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

Component Interactions

• Host → Enclave Commands: Pipeline CLI sends commands via VSOCK
• File Integrity: fs-monitor watches files, publishes hashes to NATS
• Attestation: ra-web-srv generates NSM attestation documents with VRF proofs
• Network Access: PF-Proxy bridges TCP traffic over VSOCK

---

Quick Start

Prerequisites

• EC2 Instance: Must be a Nitro Enclave-enabled instance type (e.g., c5.xlarge, m5.2xlarge)
• Operating System: Amazon Linux 2023 (recommended) or Amazon Linux 2
• Rust: Version 1.91.1 or later
• Docker: For build environment isolation

1. System Setup

# Clone the repository
git clone https://github.com/sentient-agi/sentient-enclaves-framework.git
cd sentient-enclaves-framework/rbuilds
Install Nitro Enclaves CLI and allocate resources
./rbuilds.sh --cmd "make_nitro"
Reboot to apply Nitro Enclaves allocator changes
sudo reboot

2. Build Everything

# Full automated build (kernel, apps, init, EIF image)
mkdir -vp ./eif/
./rbuilds.sh --tty --debug --network --init-c --cmd "make_all" 2>&1 3>&1

3. Run the Enclave

# Launch enclave in debug mode with attached console
./rbuilds.sh --tty --debug --network --init-c --cmd "runeifimagedebugmodecli" 2>&1 3>&1

4. Interact with the Enclave

# From host: Execute command inside enclave
./pipeline run -- ls -la /apps/
Send file to enclave
./pipeline send-file ./local-data.txt /apps/data.txt
Receive file from enclave
./pipeline recv-file /apps/results.txt ./local-results.txt

---

Components

Pipeline - Secure Local Channel Protocol

Location: pipeline/

The Pipeline component provides secure, encrypted communication between host and enclave via VSOCK.

Features

• Command execution inside enclave
• Bidirectional file transfer
• Cryptographic channel security
• Asynchronous operation modes

Usage

# Start server (inside enclave)
pipeline listen --port 53000
Execute command (from host)
pipeline run --cid 127 --port 53000 -- /usr/bin/app --flag value
File operations
pipeline send-file --cid 127 --port 53000 ./input.bin /apps/input.bin
pipeline recv-file --cid 127 --port 53000 /apps/output.bin ./output.bin

Configuration

File: .config/pipeline.config.toml

cid = 127
port = 53000

---

Remote Attestation Web Server (ra-web-srv)

Location: ra-web-srv/

HTTPS REST API for remote attestation services, providing cryptographic proofs of enclave integrity.

Features

• SHA3-512 file hashing
• VRF (Verifiable Random Function) proofs
• NSM attestation document generation
• Certificate chain validation
• NATS JetStream persistence

API Endpoints

| Endpoint | Method | Description | |----------|--------|-------------| | /generate | POST | Start processing files/directories | | /hash/ | GET | Get SHA3-512 hash for file | | /proof/ | GET | Get VRF proof for file | | /doc/ | GET | Get attestation document | | /pcrs/ | GET | Get enclave PCR registers | | /verify_hash/ | POST | Verify file hash | | /verify_proof/ | POST | Verify VRF proof | | /verify_doc/ | POST | Verify attestation document signature | | /verifycertbundle/ | POST | Verify certificate chain | | /pubkeys/ | GET | Get server public keys | | /nsm_desc | GET | Get NSM device description | | /rng_seq | GET | Get cryptographic random bytes |

Example Usage

# Generate attestation for directory
curl -k -X POST https://127.0.0.1:8443/generate \
  -H "Content-Type: application/json" \
  -d '{"path": "/apps/data"}'
Get file hash
curl -k "https://127.0.0.1:8443/hash/?path=/apps/data/file.txt"
Get attestation document
curl -k "https://127.0.0.1:8443/doc/?path=/apps/data/file.txt&view=json_hex"
Verify PCRs against expected values
curl -k -X POST https://127.0.0.1:8443/verify_pcrs/ \
  -H "Content-Type: application/json" \
  -d '{"pcrs": "0: abc123...\n1: def456..."}'

Configuration

File: .config/rawebsrv.config.toml

[ports]
http = 8080
https = 8443
[keys] sk4proofs = ""  # Auto-generated if empty sk4docs = ""
vrfciphersuite = "SECP256R1SHA256TAI"
[nats] natspersistencyenabled = 1 nats_url = "nats://127.0.0.1:4222" hashbucketname = "fs_hashes" attdocsbucketname = "fsatt_docs" persistentclientname = "rawebsrv"

---

Port Forwarding Proxy (pf-proxy)

Location: pf-proxy/

Bidirectional network proxy bridging TCP connections over VSOCK.

Binaries

| Binary | Direction | Description | |--------|-----------|-------------| | ip-to-vsock | Host → Enclave | Forward host TCP to enclave VSOCK | | vsock-to-ip | Enclave → Host | Forward enclave VSOCK to host TCP | | ip-to-vsock-transparent | Host → Enclave | Transparent TCP forwarding | | vsock-to-ip-transparent | Enclave → Host | Transparent VSOCK forwarding | | transparent-port-to-vsock | Host → Enclave | Port-based transparent proxy |

Usage

# Host side: Forward local port 8080 to enclave
ip-to-vsock --listen-ip 0.0.0.0 --listen-port 8080 \
            --vsock-cid 127 --vsock-port 8080
Enclave side: Forward VSOCK to internal service
vsock-to-ip --vsock-port 8080 \
            --target-ip 127.0.0.1 --target-port 8080

---

File System Monitor (fs-monitor)

Location: fs-monitor/

Real-time file system integrity monitoring using inotify events.

Features

• Recursive directory monitoring
• Pattern-based file exclusion (.fsignore)
• SHA3-512 hashing
• NATS KV persistence
• Debounced event processing

Usage

fs-monitor --directory "/apps/" \
           --ignore-file ".fsignore" \
           --nats-url "nats://127.0.0.1:4222" \
           --kv-bucket-name "fs_hashes"

Ignore File Format

File: .fsignore

# Comments start with #
*.log
*.tmp
.git/
node_modules/
target/

---

Enclave Init System (enclave-init)

Location: enclave-init/

Lightweight PID 1 process for managing services inside the enclave.

Features

• Service dependency resolution (before, after, requires)
• Automatic service restart policies
• Signal handling (SIGCHLD, SIGTERM, SIGHUP)
• Service logging with rotation
• Control socket (Unix & VSOCK)
• Process management

Service File Format

Location: /etc/init.d/*.service

[Service]
ExecStart = "/apps/my-service --config /apps/.config/service.toml"
Environment = ["LOG_LEVEL=info", "PORT=8080"]
WorkingDirectory = "/apps"
Restart = "on-failure"
RestartSec = 5
ServiceEnable = true
Dependencies
After = ["network.service", "nats.service"]
Before = []
Requires = ["nats.service"]
RequiredBy = []

Restart Policies

| Policy | Description | |--------|-------------| | no | Never restart | | always | Always restart regardless of exit code | | on-failure | Restart only on non-zero exit code | | on-success | Restart only on zero exit code |

Control Interface

The init system exposes a JSON-RPC control interface:

# Via Unix socket
echo '{"ServiceList":{}}' | nc -U /run/init.sock
Via VSOCK from host
echo '{"ServiceStart":{"name":"myservice"}}' | nc vsock:3:1234

Available Commands

• Ping / Pong
• ListServices
• ServiceStatus { name }
• ServiceStart { name }
• ServiceStop { name }
• ServiceRestart { name }
• ServiceEnable { name }
• ServiceDisable { name }
• ServiceLogs { name, lines }
• ProcessList
• ProcessStart { command, args, env }
• ProcessStop { pid }
• SystemStatus
• SystemReload
• SystemShutdown

---

Enclave Engine

Location: enclave-engine/

High-level enclave provisioning and management API.

Features

• Enclave lifecycle management
• Configuration-based provisioning
• Nitro CLI integration
• Multi-enclave support

Configuration

File: config.yaml

enclaves:
  default:
    cpu_count: 4
    memory_mb: 4096
    eif_path: "./enclave.eif"
    debug_mode: false
    cid: 127

---

Reproducible Builds Build System (rbuilds)

Location: rbuilds/

Comprehensive build automation system for reproducible enclave images.

Build Stages

| Stage | Command | Description | |-------|---------|-------------| | Nitro Setup | make_nitro | Install Nitro Enclaves CLI, configure allocator | | Kernel | make_kernel | Build custom Linux kernel with NSM support | | Apps | make_apps | Build Rust applications (Pipeline, ra-web-srv, etc.) | | Init | make_init | Build init system (C, Go, or Rust variants) | | EIF | make_eif | Assemble Enclave Image Format file | | All | make_all | Full automated build pipeline |

CLI Reference

# Full syntax
./rbuilds.sh [OPTIONS] --cmd "COMMAND"
Options
--tty                    # Allocate TTY for interactive output
--debug                  # Enable verbose logging
--network                # Enable both forward and reverse proxy
--rev-net                # Enable reverse proxy only
--fw-net                 # Enable forward proxy only
--init-c                 # Use C init system
--init-go                # Use Go init system
--init-rs                # Use Rust init system
--kernel VERSION         # Specify kernel version (default: 6.14.5)
--memory SIZE            # Enclave memory in MiB (default: 838656)
--cpus COUNT             # Enclave CPU count (default: 64)
--cid VALUE              # VSOCK CID (default: 127)
--dockerfile FILE        # Custom dockerfile for rootfs

Example Build Workflows

# Build with networking support
./rbuilds.sh --tty --debug --network --init-c \
  --dockerfile ./pipeline-slc-network-al2023.dockerfile \
  --cmd "make_all" 2>&1 3>&1
Build individual stages
./rbuilds.sh --tty --debug --cmd "make_kernel" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_apps" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_init" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_eif" 2>&1 3>&1
Enclave management
./rbuilds.sh --cmd "runeifimagedebugmodecli" 2>&1 3>&1
./rbuilds.sh --cmd "list_enclaves" 2>&1 3>&1
./rbuilds.sh --cmd "attachconsoleto_enclave" 2>&1 3>&1
./rbuilds.sh --cmd "drop_enclave" 2>&1 3>&1
./rbuilds.sh --cmd "dropenclavesall" 2>&1 3>&1
Cleanup
./rbuilds.sh --cmd "make_clear" 2>&1 3>&1

Interactive Mode

# Start interactive shell
./rbuilds.sh
Commands at prompt
> help              # Show available commands
> make all          # Build everything
> list_enclaves     # List running enclaves
> q                 # Toggle confirmation prompts
> lsh               # Enable local shell commands
> exit              # Exit shell

Automation Interface

# Pipe commands for automation
{ echo "makekernel"; echo "makeapps"; } | ./rbuilds.sh 2>&1
With logging
{ echo "make all"; } | /usr/bin/time -v -o ./build.log ./rbuilds.sh 2>&1 | tee ./build.output

---

Configuration

Directory Structure

/apps/
├── .config/
│   ├── pipeline.config.toml
│   ├── rawebsrv.config.toml
│   └── nats.config
├── .logs/
├── certs/
│   ├── cert.pem
│   └── skey.pem
├── pipeline
├── ra-web-srv
├── fs-monitor
├── nats-server
├── init.sh
└── .fsignore

Environment Variables

| Variable | Description | Default | |----------|-------------|---------| | CERT_DIR | TLS certificate directory | ./certs/ | | INIT_CONFIG | Init system config path | /etc/init.yaml | | LOG_LEVEL | Logging verbosity | info |

---

Security

Attestation Flow

• File Hashing: SHA3-512 hash computed for target files
• VRF Proof Generation: Deterministic proof using EC-VRF
• NSM Attestation: Hardware-rooted attestation document from Nitro Security Module
• Certificate Validation: Full X.509 chain verification against AWS root CA

PCR Registers

| PCR | Description | |-----|-------------| | PCR0 | Enclave image hash | | PCR1 | Linux kernel and bootstrap hash | | PCR2 | Application hash | | PCR3 | IAM role hash (if applicable) | | PCR4 | Parent instance ID | | PCR8 | Signing certificate hash |

Supported Cipher Suites (VRF)

• SECP256R1SHA256TAI (default)
• SECP384R1SHA384TAI
• SECP521R1SHA512TAI
• BRAINPOOLP256R1SHA256_TAI
• And more...

---

Troubleshooting

Common Issues

"Missing configuration file" Error

# Create required config directories
mkdir -p .config/
cp pipeline/.config/pipeline.config.toml .config/

VSOCK Connection Refused

# Verify enclave is running
nitro-cli describe-enclaves
Check Pipeline server is listening
./rbuilds.sh --cmd "attachconsoleto_enclave"
Look for "Pipeline listening on port 53000"

Enclave Won't Start

# Check allocator status
systemctl status nitro-enclaves-allocator
Verify resource allocation
cat /etc/nitro_enclaves/allocator.yaml
Check for enough memory/CPUs
free -h
nproc

PCR Mismatch

# Get expected PCRs from EIF build
cat ./eif/app-builder-secure-enclaves-framework.eif.pcr
Compare with running enclave
curl -k https://127.0.0.1:8443/pcrs/

Debug Mode

# Run enclave with debug console
./rbuilds.sh --debug --cmd "runeifimagedebugmodecli"
View enclave debug output
nitro-cli console --enclave-name appbuildersecureenclavesframework_toolkit

Logs

# Build logs
cat ./eif/make_build.log
cat ./eif/run-enclave.log
Inside enclave
cat /apps/.logs/pipeline.log
cat /apps/.logs/ra-web-srv.log
cat /apps/.logs/fs-monitor.log
cat /apps/.logs/nats-server.log

---

Development

Building from Source

# Build all crates
cargo build --release
Run tests
cargo test --all
Build specific component
cargo build --release -p pipeline
cargo build --release -p ra-web-srv
cargo build --release -p fs-monitor
cargo build --release -p enclave-init

Project Layout

secure-enclaves-framework/
├── pipeline/           # Secure channel protocol
├── pf-proxy/           # Port forwarding proxies
├── ra-web-srv/         # Remote attestation server
├── fs-monitor/         # File system monitoring
├── enclave-init/       # Init system (Rust)
├── enclave-engine/     # Enclave management API
├── rbuilds/            # Build system
│   ├── rbuilds.sh      # Main build script
│   ├── init_apps/      # Init source (C, Go)
│   ├── kernel_config/  # Kernel configurations
│   ├── enclave.init/   # Runtime scripts
│   └── *.dockerfile    # Build dockerfiles
└── docs/               # Documentation

---

License

This project is licensed under the Apache 2.0 License. See the LICENSE-APACHE file for the details.

---

Support

For issues, questions, or contributions, please refer to the project repository:

• Issue Tracker: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: GitHub Wiki
• Email: Sentient Enclaves Team

---

Contributing

Contributions are welcome!

Getting Started

Please follow these guidelines:

• Fork the repository
• Create a feature branch (git checkout -b feature/feature-name)
• Make your changes
• Add tests
• Run tests: cargo test
• Run lints: cargo clippy
• Commit changes (git commit -am 'Add feature')
• Push to branch (git push origin feature/feature-name)
• Submit a Pull Request

Code Style

• Follow Rust naming conventions
• Use rustfmt for formatting
• Add documentation for public APIs and new features
• Include tests for new functionality

Please ensure:

• Code follows Rust style guidelines (rustfmt)
• All tests pass
• New features include documentation
• Security implications are considered

Pull Request Guidelines

• Clear description of changes
• Link to related issues
• Include test coverage
• Update documentation as needed

---

Acknowledgments

• AWS Nitro Enclaves team for the underlying TEE infrastructure
• The Rust community for excellent cryptographic libraries
• Contributors to the VRF, COSE, and attestation standards

---

Further Reading

• AWS Nitro Enclaves Documentation
• VSOCK Protocol
• COSE (RFC 8152)
• VRF (RFC 9381)

---