You do not need a cloud account, a credit card, or a tower of graphics cards to build a working agent. A laptop, a small open-weight model and roughly forty lines of Python will get you something that reads a request, decides to use a tool, looks at what came back, and keeps going until the job is finished.
The build takes about half an hour. The trick is to stop treating the model as the product. The model is one component inside a loop: it picks the next action, your code performs it, and the result goes back into the conversation. Tools, memory and guardrails are plumbing you own and control.
What you are actually building
A plain chatbot takes a message and returns text. An agent takes a goal, chooses an action, observes the outcome, and repeats until it can stop. Four pieces make that work:
- The model — it returns structured output instead of prose when an action is needed.
- The loop — ordinary code that runs tool calls and feeds results back in.
- Tools — plain functions: read a file, do arithmetic, query a database, call an API.
- Memory — the running conversation for the current task, plus somewhere durable to keep notes.
Keep version one tiny. Two tools and a list of messages are enough to prove the pattern. Be aware that small local models are noticeably worse at structured output than the large hosted ones, so plan for the occasional malformed reply from the start rather than bolting on error handling later.
Step 1: Run a local model with Ollama
Ollama is the shortest path to a local model you can call over HTTP. Install it, then open a terminal and pull a small model:
ollama pull qwen2.5:7b
Test that it answers before you write any agent code:
ollama run qwen2.5:7b "Reply with the single word: ready"
Quantized models in the 3B to 8B range typically fit in 8GB to 16GB of memory. If the machine starts swapping or replies crawl, step down to a 3B model — for a first agent, speed matters more than polish. Ollama serves an endpoint on localhost (port 11434 by default) and ships a Python client, so you can talk to it from a script within minutes.
One small tip: set a generous keep-alive so the weights stay loaded between calls. Reloading the model on every request is the most common reason a prototype feels unusably slow.
Step 2: Set up the project
- Make a folder and a virtual environment: python -m venv .venv, then activate it.
- Install what you need: pip install ollama, plus any library your tools depend on.
- Create a single file, agent.py. You can split it into modules later.
Inside that file, write one function that takes a list of messages and returns the model's reply. Messages are dictionaries with a role (system, user, assistant) and content. That list is the agent's entire working state, which is worth internalising now: everything else you add is either a new message or a way to produce one.
Step 3: Give it two tools
A tool is a function plus a description the model can read. Start with something harmless and easy to verify: a file reader and a calculator.
Describe both tools in your system prompt as JSON objects with a name, a purpose and a list of arguments. Then instruct the model to reply either with a normal answer or with a single JSON object naming the tool and its arguments — for example, a tool called read_file with an argument path set to notes.txt.
Your code then does three things. It tries to parse the reply as JSON. If that works, it runs the matching function from a dictionary of allowed tools and appends the result to the message list as a tool result. Then it calls the model again with the updated list. If parsing fails, reply that the output was not valid JSON and give it one more attempt. After two failures, stop and ask the user to rephrase.
Making tool calls reliable
Local models drift, especially in long conversations. These habits keep them in line:
- Put the schema in the system prompt and repeat it immediately before the model has to choose.
- Keep names and arguments short and literal. read_file(path) beats performFilesystemOperation.
- Set temperature low — around 0 to 0.2 — for the decision step.
- Validate arguments before executing: does the file exist, is the value actually a number.
- During development, log every raw reply. Most failures are formatting problems, not reasoning problems.
Step 4: Add memory in two layers
Short-term memory is the message list you already have, and it has two failure modes: it grows without limit, and it eventually crowds the instructions out of the model's attention. Cap it. Keep the system prompt plus the last several exchanges, and when you trim, summarise the older turns into a few lines and keep that summary instead of the raw text.
Long-term memory is just searchable storage. When a task finishes, write a short record to a JSON file or a SQLite table: a timestamp, the goal, and what actually worked. At the start of a session, retrieve the few most relevant records and add them to the prompt. Keyword matching is fine for a first pass. Embeddings with a library such as sentence-transformers or Chroma make retrieval fuzzier and more forgiving, but do not reach for a vector database on day one — it adds moving parts without teaching you anything new about the loop.
Step 5: Add guardrails before features
An agent that can write files can overwrite them. Cap the loop at around ten iterations so a confused model cannot spin forever. Maintain an allowlist of tools and reject anything not on it. Give write and delete tools a dry-run mode for the first week, where they print what they would do instead of doing it. Keep an append-only log of every call, its arguments and its result; when something breaks at eleven at night, that log is the only useful evidence you will have. Finally, never hand an agent credentials it does not strictly need, and treat any tool that touches money, customer data or production systems as something that requires your explicit approval each time.
Your first run, and what comes next
Give the agent a task with a clear finish line: "Read every .txt file in ./inbox, pull out the names and dates, and write them to a CSV." Watch the loop run in your terminal. Expect it to fumble once or twice — that is the point. You will see precisely which layer failed: the prompt, the schema, your tool code, or the model itself. Each of those has a different fix.
Then pick one boring task you repeat every week and point the agent at it. Add a single tool, run it, fix what breaks, and repeat. After a few rounds you will have something you genuinely use, plus a clear sense of where a compact local model is sufficient and where you need a larger one. Keep the code boring and the tools narrow. That combination survives contact with real work far better than a clever architecture you cannot debug.
Photo: StartupStockPhotos / Pixabay
