✅ Open Source Foundation: Built upon Llama.cpp / llama-cpp-python and Gradio , under MIT license, Samantha runs on standard computers, even without a dedicated Graphics Processing Unit (GPU).

✅ Offline Capability: Samantha operates independently of the internet, requiring connectivity only for the initial download of model files, for executing code that requires internet access, or for using an online text-to-speech synthesizer. This ensures privacy and security for your data processing needs. Your sensitive data is not shared via the internet with companies through confidentiality agreements.

✅ Unlimited and Free Use: Samantha's open source nature allows for unrestricted use without any costs or limitations, making it accessible to anyone, anywhere, anytime.

✅ Extensive Model Selection: With access to thousands of foundation and fine-tuned gguf open source models, users can experiment with various AI capabilities, each tailored to different tasks and applications, allowing to chain the sequence of models that best meet your needs.

✅ Copy and Paste LLM: To try out a sequence of gguf models, simply copy their download links from any Hugging Face repository and paste them into Samantha to run them sequentially.

✅ Customizable Parameters: Users have full control over model hyperparameters such as context window length (nctx, maxtokens), token sampling (temperature, tfsz, top-k, top-p, minp, typicalp), penalties (presencepenalty, frequencypenalty, repeatpenalty) and stop words (stop), allowing for responses that suit specific requirements, with deterministic or stochastic behavior.

✅ Random Hyperparameter Adjustments: You can test random combinations of hyperparameter settings and observe their impact on the responses generated by the model.

✅ Interactive Experience: Samantha's chaining functionality enables users to generate endless texts by chaining prompts and models, facilitating complex interactions between different LLMs without human intervention.

✅ Feedback Loop: This feature allows you to capture the response generated by the model and feed it back into the next cycle of the conversation.

✅ Prompt List: You can add any number of prompts (separated by $$$\n or \n) to control the sequence of instructions to be executed by the models. It is possible to import a TXT file with a predefined sequence of prompts.

✅ Model List: You can select any number of models and in any order to control which model responds to the next prompt.

✅ Cumulative Response: You can concatenate each new response by adding it to the previous response to be considered when generating the next response by the model. It is important to highlight that the set of concatenated responses must fit in the model's context window.

✅ Learning Insights: A feature called Learning Mode lets users observe the model's decision-making process, providing insights into how it selects output tokens based on their probability scores (logistic units or just logits) and hyperparameter settings. A list of the least likely selected tokens is also generated.

✅ Voice Interaction: Samantha supports simple voice commands with offline speech-to-text Vosk (English and Portuguese) and text-to-speech with SAPI5 voices, making it accessible and user-friendly.

✅ Audio Feedback: The interface provides audible alerts to the user, signaling the beginning and end of the text generation phase by the model.

✅ Document Handling: The system can load small PDF and TXT files. User prompts, system prompt and model's URL list can be inputted via a TXT file for convenience.

✅ Versatile Text Input: Fields for prompt insertion allow users to interact with the system effectively, including system prompt, previous model response and user prompt to guide the model's response. You can save copied texts to TXT files.

✅ Code Integration: Automatic extraction of Python code blocks or HTML code from model's response, along with pre-installed JupyterLab integrated development environment (IDE) in an isolated virtual environment, enables users to run generated code swiftly for immediate results.

✅ Edit, Copy and Run Python and HTML Code: The system allows the user to edit the code generated by the model and run it by selecting, copying with CTRL + C and clicking the Run code button. You can also copy a Python or HTML code from anywhere (e.g. from a webpage or generated by an online AI system like ChatGPT, Claude, Gemini or DeepSeek) with CTRL + C and run it just by pressing the Run code button (as long as the code uses the installed Python libraries).

✅ Code Blocks Editing: Users can select and run Python code blocks generated by the model that uses the libraries installed in the jupyterlab virtual environment by entering the #IDE comment in the output code, selecting and copying with CTRL + C, and finally clicking the Run code button;

✅ HTML Output: Display Python interpreter output in an HTML pop-up window when text printed in the terminal is other than '' (empty string). This feature allows, for example, to execute a script unlimitedly and only display the result when a certain condition is met;

✅ Automatic Code Execution: Samantha offers the option to automatically run the Python and HTML code generated by the models in sequence. The Python code is executed by a Python interpreter within a virtual environment that includes various libraries, providing an intelligent agent-like capability. The HTML code is displayed in a new browser tab.

✅ Stop Condition: Stops Samantha if the automatic execution of the Python code generated by the model prints in the terminal a value other than '' (empty string) and that does not contain error message. You can also force exit a running loop by creating a function that returns only the string STOP_SAMANTHA when a certain condition is met.

✅ Incremental Coding: Using deterministic settings, create Python code incrementally, making sure each part works before moving on to the next.

✅ Complete Access and Control: Through the ecosystem of Python libraries and the codes generated by the models, it is possible to access computer files, allowing you to read, create, change and delete local files, as well as access the internet, if available, to upload and download information and files.

✅ Keyboard and Mouse Automation: You can create a sequence of prompts to automate tasks on your computer using the PyautoGUI library (see Automate the Boring Stuff with Python. You can even convert Python files (.py) to executable files (.exe) using the Auto-Py-To-Exe button, a Graphical User Interface (GUI) for the Pyinstaller library.

✅ Data Analysis Tools: A suite of data analysis tools like Pandas, Numpy, SciPy, Scikit-Learn, Matplotlib, Seaborn, Vega-Altair, Plotly, Bokeh, Dash, Streamlit, Ydata-Profiling, Sweetviz, D-Tale, DataPrep, NetworkX, Pyvis, Selenium, Playwright, Pytesseract, PyMuPDF, SQLAlchemy, Pygame and Beautiful Soup are available for comprehensive analysis and visualization. Integration with DB Browser (see DB Browser button) and Tesseract OCR for Windows (installation of executable file required. Provide the full path to the .exe file) is also available.
For a complete list of all Python libraries intalled in jupyterlab virtual environment, use a prompt like "Create a Python code that prints all modules installed using pkgutil library." and press Copy Code and Run Code buttons after code generation. The result will be displayed in a browser popup. You can also use pipdeptree --packages module_name in any environment-enabled terminal to see its dependencies, and pip check to check conflicts.

✅ Automated Workflow Creation: The system allows automatically saving copied text as a Python file (.py) and sequentially executing multiple Python files according to their numbering. Simply copy the code with CTRL + C, click the Save Python File button and specify a numeric prefix for the file (e.g., 01setup.py, 02process.py). When ready, the user can select all saved files and click the Run Files button to run the files in the correct numerical order, creating modular and organized workflows.

✅ Performance Optimized: To ensure smooth performance on CPUs, Samantha maintains a limited chat history to just the previous response, reducing the model's context window size to save memory and computational resources.

Installable version:

You can also install Samantha using the following procedure:

1) Install Microsoft Visual Studio Community (free community version) on your computer. Download it, run it, and select only the option Desktop development with C++ (administrator privileges required):

2) Download the zip file from Samantha's repository by clicking here and unzip it to your computer. Select the drive where you want to install the program (Ex.: C:\Users\You\Documents):

3) Open samanthaia-main directory and double click on installsamanthaia.bat file to start installation. Windows may ask you to confirm the origin of the .bat file. Click on 'More info' and confirm. We encorage to inspect the code of all files (use VirusTotal and AI systems to do so):

2) Visit Hugging Face repository and click on the card to open the corresponding page. Locate the Files and versions_ tab and choose a .gguf text generation model that fits in your available memory. 3) Right click over the model download link icon and copy its URL.

4) Paste the model URL into Samantha's Download models for testing field.

5) Insert a prompt into User prompt field and press Enter. Keep the $$$ sign at the end of your prompt. The model will be downloaded and the response will be generated using the default deterministic settings. You can track this process via Windows Task Management.

Every new model downloaded via this copy and paste procedure will replace the previous one to save hard drive space. Model download is saved as MODELFORTESTING.gguf in your Downloads folder.

You can also download the model and save it permanently to your computer. For more datails, see the section below.

Downloading Open Source Model Files (.gguf)

Open souce text generation models can be downloaded from Hugging Face, using gguf as the search parameter. You can combine two words like gguf code or gguf portuguese.

You can also go to a specific repository and see all the .gguf models available for downloading and testing, like https://huggingface.co/bartowski or https://huggingface.co/NousResearch.

The models are displayed on cards like this:

To download the model, click on the card to open the corresponding page. Locate the Model card and Files and versions tabs:

To download some models, you must agree to the terms of use.

After that, click on the Files and versions tab and download a model that fits in your available RAM space. To check your available memory, open Windows Task Manager by pressing CTRL + SHIFT + ESC, click on Performance tab (1) and select Memory (2):

We suggest to download the model with Q4KM (4-bit quantization) in its link name (put the mouse over the download button to view the complete file name in the link like this: https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF/resolve/main/Hermes-2-Pro-Llama-3-8B-Q4KM.gguf?download=true). As a rule, the larger the model size, the greater the accuracy of the generated text.

If the downloaded model doesn't fit into the available RAM space, your hard drive will be used, impacting performance.

Download the chosen model and save it to your computer or just copy the download link and paste it into Samantha's Download model for testing field.

Note that each model has its own characteristics, presenting significantly different responses depending on its size, internal architecture, training method, predominant language of the training database, user prompt and hyperparameter adjustment, and it is necessary to test its performance for the desired task.

Some models may not be loaded due to their technical characteristics or incompatibility with the current version of the llama.cpp Python binding used by Samantha.

Where to find models to test: Huggingface GGUF Models

Samantha is an experimental program, created to test open source AI models. Therefore, it is common for errors to occur when trying to test a new model or new versions of models created by users.

The quality of the responses generated by a model can be evaluated using some criteria, such as:

* Degree of understanding of the explicit and implicit instructions contained in the user and system prompts;

* Degree of obedience to these instructions, aspect related to the predominant language of the database;

* Degree of hallucination in the generation of coherent text, but incorrect or out of context. Hallucination in text generation typically results from insufficient training of the model, resulting in a inappropriate selection of the next token, which leads the model in an undesired semantic direction;

* Degree of precision in the decision-making process to fill in the gaps in the context of the user prompt and to resolve ambiguities necessary to generate the response. What is not explicitly specified, the model tries to infer based on its training, which can lead to errors;

* Degree of coherence of the bias adopted by the model with the bias (or lack thereof) contained in the user's prompt;

* Degree of pertinence and relevance of the topics choosen to be addressed;

* Degree of breadth and depth of approach to topics in the response;

* Degree of syntatic and semantic precision of the response;

* Quality of the structure and content of the response in relation to the user's expectations (and their overcoming) for the problem submitted to the model, considering the technique used to create the prompt (prompt engineering) and the adjustment of the model's hyperparameters.

This button also clears the internal previous response (not the text in the Assistant previous response field).

A chat session can contain more than one conversation cycle (loop).

Start chat button keyboard shortcut: Press Enter anywhere on the page.

To generate text, a model must be pre-selected in Model selection dropdown list. You can download .gguf models from the Hugging Face repository.

Alternatively, a Hugging Face model URL can be provided into Download model for testing field.

If both fields are filled in, the model selected via the dropdown list takes precedence.

When this button is pressed, the system starts loading the model selected in the dropdown list or downloading the model whose link was entered in the Download model for testing field. The outline of the Output field starts flashing slowly in orange to indicate the start of the process.

If the model is already loaded into memory, pressing the button starts generating the text.

Press CTRL + SHIFT + ESC to open Windows Task Manager and monitor the process.

🛑​ Interrupts the token generation process for the current model or prompt, starting the execution of the next model or prompt in the sequence, if any.

It also stops playback of the currently playing audio when in speech autoplay mode (Read response aloud checkbox selected).

Samantha has 3 phases:
1) Loading model (non stop) 2) Thinking (non stop) 3) Next token selection (stop).

This button interrupts the token generation only when the next token selection phase is started, even if it was pressed previously.

This interruption does not prevent the execution of the code generated by the model, if the Run code automatically checkbox is selected. You can press the button to stop text generation and run the already generated Python code.

🧹​ Clears the history of the current chat session, erasing the assistant output field as well as all internal logs, previous response etc.

For this button to work, you need to wait for the model to finish generating the text (orange border of the Assistant output field stops blinking)

✅​ Allows you to select the directory where the models available for loading are saved.

Default: Windows "Downloads" folder

You can select any directory that contains GGUF models. In this case, the models contained in the selected directory will be listed in the Model selection dropdown list.

When the pop-up window opens, make sure to click on the folder you want to select.

🛑 Stops the sequence of running models and resets internal settings of the last loaded model.

After resetting, models take some time to restart text generation, depending on the size of the input text.

This interruption prevent the execution of the already Python code generated by the model, if the Run code automatically option is selected.

📑 Replaces the text in the Assistant previous response field with the text of the last response generated by the model.

The replaced text will be used as the model's previous response in the next conversation cycle.

This replaced text is not visible. It does not erase text from the Previous Assistant Response field, which can be used again later.

💻​ In the context of Large Language Models (LLMs), a system prompt is a special type of instruction given to the model at the beginning of a conversation or task. It is considered in all interactions with the model.

Think of it as setting the stage for the interaction. It provides the LLM with crucial information about its role, the desired persona, behavior, and the overall context of the conversation.

Here's how it works:

• Defining the Role: The system prompt clearly defines the LLM's role in the interaction.

• Setting the Tone and Persona: The system prompt can also establish the desired tone and persona for the LLM's responses.

• Providing Contextual Information: The system prompt can offer background information relevant to the conversation or task.

Benefits of Using System Prompts:

• Improved Consistency: System prompts ensure that the LLM consistently adheres to a specific role and style throughout the interaction.
• Enhanced Accuracy: By providing context and instructions, system prompts help the LLM generate more accurate and relevant responses.
• Tailored Experiences: Different system prompts can be used to create tailored experiences for users based on their needs and preferences.

Let's say you want to use an LLM to write a poem in the style of Shakespeare. A suitable system prompt would be:

You are William Shakespeare, a renowned poet from Elizabethan England.

By providing this system prompt, you guide the LLM to generate a response that reflects Shakespeare's language, style, and thematic interests.

Not all models support system prompt. Test to find out: fill in "x = 2" in the System prompt field and ask the model the value of "x" in the User prompt field. If the model gets the value of "x", system prompt is available in the model.

You can simulate the effect of the system prompt by adding text in square brackets in the beginning of the User prompt field:

[This text acts as a system prompt]

To ignore the text present in this field, include --- at the beginning. To split the text in parts, put $$$ between them. To ignore each part, include --- at the beginning of each part.

↩️​ When activated, it automatically considers the response generated by the model in the current conversation cycle as being the Assistant's previous response in the next cycle, allowing feedback from the system.

Any text entered by the user in the Assistant previous response field is only considered in the first cycle after activating this feature. In the following cycles, the model's response internally replaces the previous response, but without deleting the text contained in that field, which can be reused in a new chat session. You can monitor the content of the assistant previous response via terminal.

In turn, when deactivated, it always uses the text contained in the Assistant previous response field as the previous response, unless the text is preceded by --- (triple dash). Text preceded by --- is ignored by the model.

To internally clear the model's previous response, press the Clean history button.

➡️​ Stores the text considered by the model as its previous response in the current conversation cycle.

Used to feed back the responses generated by the model.

To ignore the text present in this field, include --- at the beginning. To split the text in parts, put $$$ between them. To ignore each part, include --- at the beginning of each part.

✏️​ The main input field of the interface. It receives the list of user prompts that will be submitted to the model sequentially.

Each item in the list must be separated from the next one by a line break (SHIFT + ENTER or \n) or by the symbols $$$ (triple dollar signal), if the items are made up of text with line breaks.

When present in the user prompt, the $$$ separator takes precedence over the \n separator. In other words, \n is ignored.

You can import a TXT file containing a list of prompts.

--- before a prompt list item causes the system to ignore that item.

Text positioned within single square brackets ([ and ]) is added to the beginning of each prompt list item, simulating a system prompt.

Text positioned within double square brackets ([[ and ]]) is added as the last item in the prompt list. In this case, all responses generated by the model in the current chat session are concatenated and added to the end of this item, allowing the model to analyze them together.

If the Python code execution returns only the word STOP_SAMANTHA, it stops token generation and exits the loop.

If the Python code execution returns only '' (empty string), it does not display the HTML pop-up window.

You can add specific hyperparameters before each prompt. You must use this pattern:

{maxtokens=4000, temperature=0, tfsz=0, topp=0, minp=1, typicalp=0, topk=40, presencepenalty=0, frequencypenalty=0, repeat_penalty=1}

Example:

[You are a poet that writes only in Portuguese]
Create a sentence about love
Create a sentence about life
---Create a sentence about time (this instruction is ignored)
[[Create a paragraph in English that summarizes the ideas contained in the following sentences:]]
(previous responses are concatenated here)

Model responses sequence:

"O amor é um fogo que arde no meu peito, uma chama que me guia através da vida."
"A vida é um rio que flui sem parar, levando-nos para além do que conhecemos."
Love and life are intertwined forces that shape our existence. Love burns within us like a fire, guiding us through life's journey with passion and purpose. Meanwhile, life itself is a dynamic and ever-changing river, constantly flowing and carrying us beyond the familiar and into the unknown. Together, love and life create a powerful current that propels us forward, urging us to explore, discover, and grow.

✅ ​Dropdown list of models saved on the computer and available for text generation.

To view models in this field, click the Load model button and select the folder containing the models.

The default location for saving models is the Windows Downloads directory.

You can select multiples models (even repeated) to create a sequence of models to respond the user prompts.

The last model downloaded from a URL is saved as MODELFORTESTING.gguf and is also displayed in this list.

​⬇️​ Receives a list of Hugging Face links to the models that will be downloaded and executed sequencially.

Link example:

• https://huggingface.co/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf?download=true

Only works if no model is selected in Model selection dropdown list.

​1️⃣​ Activates a single response per model.

Prompts that exceed the number of models are ignored.

Models that exceed the number of prompts are also ignored.

You can select the same model more than once.

This checkbox disables Number of loops and Number of responses checkboxes.

​​​⏮️ Reinitializes the internal state of the model, eliminating the influence of the previous context.

How it Works:

When the reset feature is invoked:

• The model's internal state related to the current context is cleared.
• Any accumulated tokens from previous interactions are discarded.
• The model essentially returns to its initial state, as if it was just loaded.

• Improved consistency: Each new interaction starts fresh, reducing the chance of the model being influenced by unrelated previous context.
• Better control: Users can manage when the model should "forget" previous interactions.

• In chatbots or conversational AI where you want to start new chat sessions cleanly.
• In applications processing multiple independent text generation tasks.
• When fine-tuning or testing the model's behavior under controlled conditions.

🎰​ Shuffles the execution order of the models if 3 or more models are selected in Model selection dropdown list.

🏃‍♀️​ Generates text faster in the background without displaying the addition of each token in the Assistant output field.

Minimizing or hiding the Samantha browser window makes the token generation process even faster.

This checkbox disables Learning Mode.

🗣️​ Selects the language of the computer's SAPI5 voice that will read the responses generated by the model.

🎶​ Activates automatic reading mode for responses generated by the model using the language selected in the Voice selection dropdown list.

If you wish to reproduce the response generated by the model using a better quality speech synthesizer (Microsoft Edge browser), open the response in an HTML pop-up using the Response in HTML button, right-click inside the page and select the option to read the page text aloud.

To save and edit the audio generated by the speech synthesizer, we recommend record de audio using the portable version of the open source program Audacity. Adjust the recording setting to capture audio output from the speakers (not from the microphone).

🎶​ Activates automatic reading mode using Edge browser's online text-to-speech.

You can stop the audio playback by pressing the Stop / Next or Stop & Reset buttons.

To save and edit the audio generated by the speech synthesizer, we recommend record de audio using the portable version of the open source program Audacity. Adjust the recording setting to capture audio output from the speakers (not from the microphone).

​👩‍🏫 ​Activates Learning Mode.

It presents a series of features that help in understanding the token selection process by the model, such as:

• Model metadata
• Tokens vocabulary
• Top-k tokens sorted by logits score with token position in the vocabulary and selected token indication
• Barplot of the top-k tokens sorted by logits scores
• Cumulative barplot of the selected unlikely tokens

Radio buttons options:

• OFF: Learning mode disabled.
  
• 0, 0.3, 1, 3, 10: Generation time delay in seconds.
  
• NEXT TOKEN: Allows you to control the response generation process, token by token, via NEXT TOKEN button.
  

🔂​ Set the number of repetitions of the block in the following chaining sequence:

Chaining Sequence: ( [models list] -> respond -> ( [user prompt list] X number of responses) ) X number of loops

Each model in the models list responds to all prompts in the user prompt list for the selected number of responses. This block is repeated for the selected number of loops.

🔂 Number of responses to be generated by each selected model in the following chaining sequence:

Chaining Sequence: ( [models list] -> respond -> ( [user prompt list] X number of responses) ) X number of loops

Each model in the models list responds to all prompts in the user prompt list for the selected number of responses. This block is repeated for the selected number of loops.

👩‍💻​ When checked, runs automatically the Python code generated by the model.

Whenever Python code returns a value other than '' (empty string), an HTML pop-up window opens to display the returned content.

​🛑​ When checked, stops Samantha when the automatic execution of the Python code generated by the model prints in the terminal a value other than '' (empty string) and that does not contain error message.

Use it to stop a generation loop when a condition is met.

​📥​ When checked, concatenates each new response by adding it to the previous response to be considered when generating the next response by the model.

It is important to highlight that the set of concatenated responses must fit in the model's context window.

🎲 Adjusts the model's hyperparameters with random values ​​in each new conversation cycle.

Randomly chosen values ​​vary within the following value range of each hyperparameter and are displayed at the beginning of each response generated by the model.

| Hyperparameter | Min. Value | Max. Value | | :-----------------: | :--------: | :--------: | | temperature | 0.1 | 1.0 | | tfsz_ | 0.1 | 1.0 | | topp_ | 0.1 | 1.0 | | minp_ | 0.1 | 1.0 | | typicalp_ | 0.1 | 1.0 | | presencepenalty_ | 0.0 | 0.3 | | frequencypenalty_ | 0.0 | 0.3 | | repeatpenalty_ | 1.0 | 1.2 |

This resource has application in the study of the reflections of the interaction between hyperparameters.

🐍 Feedback only the Python interpreter output as the next assistant's previous response. Do not include model's response.

This feature reduces the number of tokens to be inserted in the assistant's previous response in the next conversation cycle.

Works only with Feedback Loop activated.

Hide HTML model responses, including Python interpreter error messages.

📖​ nctx stands for number of context tokens_ in the context window and determines the maximum number of tokens that the model can process at once. It determines how much previous text the model can "remember" and utilize when selecting the next token from model vocabulary.

The context length directly impacts the memory usage and computational load. Longer n_ctx requires more memory and computational power.

How n_ctx works:

It sets the upper limit on the number of tokens the model can "see" at once. Tokens are usually word parts, full words, or characters, depending on the tokenization method. The model uses this context to understand and generate text. For example, if n_ctx is 2048, the model can process up to 2048 tokens (now words) at a time.

Impact on model operation:

During training and inference, the model attends to all tokens within this context window.
It allows the model to capture long-range dependencies in the text.
Larger n_ctx enables the model to handle longer sequences of text without losing earlier context.

Why increasing n_ctx increases memory usage:

Attention mechanism: LLMs uses self-attention mechanisms (like in Transformers) which compute attention scores between all pairs of tokens in the input.
Quadratic scaling: The memory required for attention computations scales quadratically with the context length. If you double n_ctx, you quadruple the memory needed for attention.

CAUTION: nctx MUST BE GREATER THAN (maxtokens + number of input tokens) (system prompt + assistant previous response + user prompt).

If the prompt text contains more tokens than the context window defined with nctx_ or the memory required exceeds the total available on the computer, an error message will be displayed.

Error message displayed on Assistant output field:

==========================================
Error loading LongWriter-glm4-9B-Q4KM.gguf.
Some models may not be loaded due to their technical characteristics or incompatibility with the current version of the llama.cpp Python binding used by Samantha.
Try another model.
==========================================

Error messages displayed on terminal:

Requested tokens (22856) exceed context window of 10016
Unable to allocate 14.2 GiB for an array with shape (25000, 151936) and data type float32

When set to 0, the system will use the maximum n_ctx possible (model's context window size).
As a rule, set nctx equal to maxtokens, but only to the value necessary to accommodate the text parsed by the model. Samantha's default values ​​for nctx and maxtokens are 4,000 tokens.
Before adjusting nctx, you must to unload the model by clicking Unload model_ button.

Example:

User prompt = 2000 tokens
n_ctx= 4000 tokens
If the text generated by the model is equals or greater than 2000 tokens (4000 - 2000), the system will raise an IndexError in the terminal, but the interface will not crash.

To check the impact of the nctx in memory, open Windows Task Manager (CTRL + SHIFT + ESC) to monitor memory usage, select memory panel and vary nctx values. Don't forget to unload model between changes.

🎚️​ Controls maximum number of tokens to be generated by the model.

Select 0 for the models' maximum number of tokens (maximum memory required).

How max_tokens Works:

• Sampling Process: When generating text, LLMs predict the next token based on the context provided (system prompt + previous response + user prompt + text already generated). This prediction involves calculating probabilities for each possible token in the vocabulary.

• Token Limit: The max_tokens parameter sets a hard limit on how