This is a proof of concept to generate Airbyte low-code YAML connectors from API documentation. We want this to serve as inspiration to what can be done with LLMs. Here's how it works:

• Specify a goal, e.g "Fetch all pages posts"
• Provide one or more links to documentation, e.g for Notion: API Intro, API Versioning and Search Endpoint

This will generate an OpenAPI specificatin and a Airbyte low-code connector. Try it!

📽️ Demo

Notion

https://github.com/skyffel/airbyte-connector-generator-poc/assets/3134895/f7e03f6d-60d9-44b6-88ee-3bc0e4bcd339

Perplexity

https://github.com/skyffel/airbyte-connector-generator-poc/assets/25622412/e2922aaa-4f19-4608-8fba-e1e08c201033

✅ Key features

• Co-pilot for generating ETL code for HTTP APIs
• Support Airbyte low-code YAML connectors
• Scrapes API documentation
• Produces OpenAPI specifications

📦 Installation

• Clone the repository

git clone https://github.com/skyffel/airbyte-connector-generator-poc
   cd airbyte-connector-generator-poc

• Setup virtual environment

# Using virtualenv
   virtualenv env
   source env/bin/activate
# Or using venv    python3 -m venv env    source env/bin/activate
# Or using poetry    poetry shell

• Install required packages

poetry install
   playwright install

🚀 How to use

Generates an Airbyte low-code YAML connector using the API documentation provided via URLs.

Set DEBUG=true in .env to enable logs

skyffel --goal "<MY ETL GOAL>" --urls "<URL DOC 1>" --urls "<URL DOC 2>"

Here we generate a connector for extracting all blog posts from the Department of Justice.

skyffel \
    --goal "extract all blog entries from department of justice" \
    --urls https://www.justice.gov/developer/api-documentation/api_v1

📥 Import to Airbyte

After generating the connector, you need to import it to Airbyte. Eventually they might expose an API to do this programatically 🤞 Until then, here's how:

• Go to your Airbyte workspace
• Click on Builder [BETA] in the menu

• Click “New custom connector” in the upper right corner

• Click “Import a YAML” and select the generated airbyte_connector.yaml

• Fill in the test values and run the test
• Press “Publish to workspace”