AISquare Studio AutoQA

ai automation testing playwright github-action crewai openai qa test-generation multi-agent

AI-powered GitHub Action that converts natural language test descriptions in pull request bodies into fully automated Playwright tests. Write what you want to test in plain English — AutoQA generates, executes, and commits production-ready test code using CrewAI multi-agent orchestration and OpenAI GPT-4.

Features

• AI-Powered Test Generation — Natural language steps become executable Playwright Python tests
• Active Execution Mode — Iterative step-by-step generation with real-time browser context
• Smart Selector Discovery — Auto-discovers optimal selectors from live pages via DOMInspectorTool
• Intelligent Retry — Automatic error recovery with alternative selectors and failure analysis
• AST-Based Security Validation — Prevents unsafe code patterns before execution
• Cross-Repository Architecture — Deploys as a GitHub Action, runs in any repository
• Comprehensive Reporting — PR comments with screenshots, HTML reports, and JSON artifacts
• ETag-Based Idempotency — Prevents duplicate test generation for unchanged PR descriptions
• Multi-Tier Test Organization — Categorize tests into A/B/C tiers by criticality
• Caching Strategy — Pip and Playwright browser caching for fast CI runs

How It Works

PR Description          AutoQA Action              Your Repository
┌──────────────┐     ┌──────────────────┐     ┌──────────────────┐
│

│     │ 5. Commit on pass│
│              │     │ 6. Comment on PR │
│  1. Go to /  │     └──────────────────┘
│  2. Login    │
│  3. Verify   │
└──────────────┘

• A developer writes numbered test steps in the PR description inside a fenced autoqa block
• The GitHub Action triggers on PR open/edit/sync events
• AutoQA parses the PR body for metadata (flow_name, tier, area) and test steps
• CrewAI agents generate Playwright Python test code from the steps
• Generated code is validated via AST analysis and executed against your staging environment
• On success, the test file is committed to tests/autoqa/{tier}/{area}/test{flowname}.py
• Results and screenshots are posted as a PR comment

Quick Start

1. Add the workflow

Create .github/workflows/autoqa.yml in your repository:

name: AutoQA Test Generation
on:   pull_request:     types: [opened, synchronize, edited]
jobs:   autoqa:     runs-on: ubuntu-latest     steps:       - uses: actions/checkout@v6         with:           token: ${{ secrets.GITHUB_TOKEN }}
- name: Generate and Execute Tests         uses: AISquare-Studio/AISquare-Studio-QA@main         with:           openai-api-key: ${{ secrets.OPENAIAPIKEY }}           staging-url: ${{ secrets.STAGING_URL }}           staging-email: ${{ secrets.STAGING_EMAIL }}           staging-password: ${{ secrets.STAGING_PASSWORD }}

2. Configure secrets

Add the following secrets in your repository's Settings → Secrets and variables → Actions:

| Secret | Description | | ------------------ | --------------------------------- | | OPENAIAPIKEY | OpenAI API key (GPT-4 access) | | STAGING_URL | Staging environment login URL | | STAGING_EMAIL | Test account email | | STAGING_PASSWORD | Test account password |

3. Write test steps in a PR

Include a fenced autoqa block in your pull request description:

markdown</code></pre>autoqa
flowname: userlogin_success
tier: A
area: auth
<pre><code class="lang-">1. Navigate to the login page
Enter valid email address
Enter valid password
Click the login button
Verify the dashboard appears</code></pre>

Open the PR and AutoQA takes care of the rest.

PR Format Reference
The autoqa code block defines metadata. Numbered steps below it describe the test scenario.
markdown</code></pre>autoqa
flowname: <snakecasetestname>
tier: <A|B|C>
area: <feature_area>
<pre><code class="lang-">1. First test step in plain English
Second test step
...</code></pre>

| Field       | Required | Description                                               |
| ----------- | -------- | --------------------------------------------------------- |
| flow_name | Yes      | Snake-case identifier used for the generated file name    |
| tier      | Yes      | A (critical), B (important), or C (nice-to-have)   |
| area      | Yes      | Feature area used as subdirectory (e.g., auth, billing) |

Configuration Reference
Action Inputs
| Input               | Required | Default          | Description                                       | | ------------------- | -------- | ---------------- | ------------------------------------------------- | | openai-api-key    | Yes  | —                | OpenAI API key
| openai-model | No | openai/gpt-4.1 | OpenAI model for test generation (e.g., openai/gpt-4.1, openai/gpt-4o) |                                   | | staging-url       | Yes  | —                | Staging environment URL                           | | qa-github-token   | No       | github.token   | GitHub token (for private repo access)             | | staging-email     | No       | test@example.com | Test account email                              | | staging-password  | No       | —                | Test account password                             | | target-repo-path  | No       | .              | Path to the target repository                     | | git-user-name     | No       | AutoQA Bot     | Git user name for test commits                    | | git-user-email    | No       | —                | Git user email for test commits                   | | pr-body           | No       | (auto-detected) | PR description text                              | | test-directory    | No       | tests/autoqa   | Base directory for generated tests                | | create-pr         | No       | false          | Create a PR for tests instead of pushing directly | | execution-mode    | No       | generate       | Execution mode: generate, suite, or all     |
Action Outputs
| Output                | Description                             | | --------------------- | --------------------------------------- | | test_generated      | Whether a test was generated (true/false) | | testfilepath      | Path to the generated test file         | | test_results        | JSON object with execution results      | | generation_metadata | JSON object with generation metadata    | | screenshot_path     | Path to captured screenshots            | | etag                | Idempotency hash of the PR description  | | flow_name           | Parsed flow name                        | | tier                | Parsed tier                             | | area                | Parsed area                             | | error               | Error message (if failed)               |
Execution Modes
| Mode       | Behavior                                                          | | ---------- | ----------------------------------------------------------------- | | generate | Parse PR, generate a new test, execute it, and commit on success  | | suite    | Run the existing test suite only (regression testing)             | | all      | Generate a new test and run the full existing suite           |

Project Structure
AISquare-Studio-QA/
├── action.yml                          # GitHub Action definition
├── qa_runner.py                        # Local test runner entry point
├── requirements.txt                    # Python dependencies
├── pyproject.toml                      # Python project configuration
├── pytest.ini                          # Pytest configuration
├── env.template                        # Environment variables template
├── .github/
│   ├── copilot-instructions.md         # Copilot custom instructions (AI agent reference)
│   └── workflows/                      # CI/CD workflows (lint, test, release)
├── config/
│   ├── autoqa_config.yaml              # AutoQA policy and settings
│   └── test_data.yaml                  # Test scenarios and selectors
├── src/
│   ├── agents/
│   │   ├── planner_agent.py            # Generates Playwright code via CrewAI
│   │   ├── executor_agent.py           # Validates and executes code (AST safety)
│   │   └── stepexecutoragent.py      # Active execution step agent
│   ├── autoqa/
│   │   ├── action_runner.py            # Main GitHub Action orchestrator
│   │   ├── parser.py                   # PR body metadata parser
│   │   ├── action_reporter.py          # PR comment generator
│   │   └── crossrepomanager.py       # Test file commits across repos
│   ├── crews/
│   │   └── qa_crew.py                  # CrewAI agent orchestration
│   ├── execution/
│   │   ├── iterative_orchestrator.py   # Step-by-step execution coordinator
│   │   ├── execution_context.py        # State tracking between steps
│   │   └── retry_handler.py            # Failure analysis and retry logic
│   ├── tools/
│   │   ├── playwright_executor.py      # Test code execution engine
│   │   └── dom_inspector.py            # Live page selector discovery
│   ├── templates/
│   │   └── testexecutiontemplate.py  # Execution template
│   └── utils/
│       ├── logger.py                   # GitHub Actions-aware logging
│       ├── githubcommentclient.py    # GitHub API client
│       ├── comment_builder.py          # Markdown comment builder
│       ├── screenshot_handler.py       # Screenshot capture
│       └── screenshotembedmanager.py # Screenshot embedding
├── tests/                              # Pytest test suites
├── docs/                               # Documentation
├── examples/                           # Example workflows and configs
├── reports/                            # Generated test artifacts
└── scripts/                            # Utility scripts

Local Development
Prerequisites
Python 3.11+
An OpenAI API key with GPT-4 access

Setup
# Clone the repository
git clone https://github.com/AISquare-Studio/AISquare-Studio-QA.git
cd AISquare-Studio-QA
Install dependencies
pip install -r requirements.txt
playwright install --with-deps chromium
Configure environment
cp env.template .env
Edit .env with your staging URL, credentials, and OpenAI API key
Running locally
# Run the test runner
python qa_runner.py
Run with visible browser for debugging
HEADLESSMODE=false python qarunner.py
Show detailed help
python qa_runner.py --help-detailed
Running the test suite
pytest tests/ -v

Architecture
AutoQA uses a multi-agent architecture powered by CrewAI:
Planner Agent — Converts natural language steps into Playwright Python code
Executor Agent — Validates generated code via AST analysis and runs it in a sandboxed browser
Step Executor Agent — Handles Active Execution Mode, processing one step at a time with live browser context

The Iterative Orchestrator coordinates step-by-step execution, maintaining state via ExecutionContext and handling failures through RetryHandler.
For a detailed architecture walkthrough, see docs/ARCHITECTURE.md.

Security
All AI-generated code is validated before execution:
AST-based validation — Blocks dangerous constructs (eval, exec, open, subprocess, file I/O)
Restricted imports — Only playwright.sync_api, time, datetime, and re are permitted
Sandboxed execution — Tests run in isolated Playwright browser contexts
Secret redaction — Sensitive values are masked in logs and reports

See the Security Model section in the architecture documentation for details.

Performance and Caching
AutoQA caches dependencies to minimize CI run times:
| Layer                 | Cache Key                       | Typical Size | | --------------------- | ------------------------------- | ------------ | | Python pip packages   | Hash of requirements.txt      | ~200 MB      | | Playwright browsers   | Playwright version              | ~100 MB      | | Action repository     | Commit SHA                      | ~5 MB        |
| Scenario  | Approximate Time | | --------- | ---------------- | | Cold run  | 3–4 minutes      | | Warm run  | 45–60 seconds    |
Caches automatically invalidate when requirements.txt changes.

Code Quality and Linting
The project enforces consistent style via automated tooling:
| Tool      | Purpose                       | Configuration          | | --------- | ----------------------------- | ---------------------- | | black | Code formatting               | Line length: 100       | | isort | Import sorting                | Black-compatible profile | | flake8| PEP 8 compliance              | Standard rules         |
The lint.yml workflow runs on every push and pull request, auto-fixing formatting issues.
# Run locally
black . --line-length=100
isort . --profile=black --line-length=100
flake8 .

Roadmap
See docs/AUTOQAENHANCEMENTROADMAP.md for the full enhancement roadmap, including 16 feature proposals inspired by Lucent AI and Meticulous AI covering AI-generated test criteria from code diffs, visual regression detection, self-healing tests, automatic bug reports, and more.

For open-source readiness status, see docs/OPENSOURCE_ROADMAP.md.

Contributing
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Fork the repository
Create a feature branch
Make your changes with tests
Ensure linting passes (black, isort, flake8)
Submit a pull request

Please review the CODEOF_CONDUCT.md before contributing.
AI agent sessions: This repository includes a
.github/copilot-instructions.md file that
GitHub Copilot reads automatically. It contains architecture reference, version
tables, and a mandatory session checklist (update CHANGELOG, README, examples, etc.).

License
This project is licensed under the Apache License 2.0.
Copyright 2025 AISquare Studio

Contributors
 | Avatar | Name | Role | | ------ | ---- | ---- | | 🤖 | AutoQA Bot | Automation | | 👩‍💻 | Zahwah | Contributor | | 👩‍💼 | Rabia | Maintainer | 

Built by AISquare Studio