All blogs

How to Build an AI Voice Agent With Python and LiveKit

A simple guide to building a real-time AI voice agent with Python and LiveKit Agents: how it works, how to build one, and what it takes to run it in production.

How to Build an AI Voice Agent With Python and LiveKit cover image

Table of Contents

  1. Introduction
  2. What is LiveKit?
  3. How a LiveKit voice agent works
  4. Building your first agent in Python
  5. Giving your agent tools
  6. Taking it to production
  7. LiveKit vs hosted voice platforms
  8. Real example: Ava, our morning brief voice agent
  9. Conclusion

TL;DR - 6 takeaways - 15-second read

  1. LiveKit Agents is an open-source framework for building real-time AI voice agents in Python.
  2. A voice agent listens (speech-to-text), thinks (an AI model), and speaks (text-to-speech), or uses a realtime model that does all three at once.
  3. Turn detection stops the agent from cutting people off in the middle of a sentence.
  4. Function tools let the agent fetch real data, like sales numbers or the weather, instead of guessing.
  5. LiveKit gives you more control than hosted platforms like Retell AI or Vapi, but it needs more developer time.
  6. Our prototype voice agent Ava uses 11 function tools to read a spoken business brief and answer follow-up questions.

Introduction

Voice AI has come a long way. A few years ago, talking to a computer meant waiting through awkward pauses and repeating yourself. Today, a good AI voice agent can hold a natural conversation, answer follow-up questions, and pull live data from your systems while you talk.

If you want to build one yourself, you have two main choices. You can use a hosted platform like Retell AI or Vapi, or you can build it with a framework like LiveKit Agents, where you write the agent in Python and control every part of it.

In this guide, we explain how LiveKit voice agents work in plain language, walk through building one in Python, and share how we built Ava, a voice agent that reads managers a daily business brief and answers their questions, all by voice.

What is LiveKit?

LiveKit is an open-source platform for real-time audio and video. It handles the hard parts of sending live voice between people and devices, like keeping latency low and dealing with bad network connections.

On top of that, LiveKit offers LiveKit Agents, a framework for building AI agents that join those live conversations. Your agent joins a "room" just like a human participant would. It listens to the user, thinks, and talks back in real time.

You can run LiveKit in two ways:

  • LiveKit Cloud. LiveKit hosts the real-time infrastructure for you, so you only write and run your agent.
  • Self-hosted. You run the LiveKit server yourself, which gives you full control over where your data goes.

According to LiveKit's own quickstart, you can have a basic voice assistant running in less than 10 minutes, and you can talk to it in your terminal, a browser, a phone call, or a mobile app (LiveKit docs).

How a LiveKit voice agent works

Every LiveKit voice agent needs to do three things: listen, think, and speak. LiveKit Agents supports two ways to do this (LiveKit docs).

Option 1: The STT-LLM-TTS pipeline

This is the most common setup. It chains three specialized models together:

PartWhat it doesSimple way to think about it
STT (speech-to-text)Turns the user's voice into textThe agent's ears
LLM (large language model)Reads the text and decides what to say or doThe agent's brain
TTS (text-to-speech)Turns the reply into a natural voiceThe agent's mouth

The big advantage is choice. You can pick the best speech model, the best AI model, and the best voice separately, and swap any of them later.

Option 2: A realtime model

Some newer models, like OpenAI's realtime models, can listen and speak directly without separate STT and TTS steps. This can feel even faster and more natural, but you have less control over each part.

Turn detection: knowing when to talk

The hardest part of a voice conversation is not understanding words. It is knowing when the other person has finished speaking. If the agent jumps in too early, it cuts people off. If it waits too long, the conversation feels slow.

LiveKit handles this with voice activity detection and a turn detector model. Together, they work out whether a pause means the person is done, or just thinking mid-sentence.

Building your first agent in Python

Here is the basic process for getting a LiveKit voice agent running. LiveKit's docs and starter project fill in every detail, so treat this as a map of the steps.

Step 1: Check your setup. LiveKit Agents needs Python 3.10 or newer. LiveKit's quickstart uses the uv package manager, but pip works too.

Step 2: Create a LiveKit project. Sign up for LiveKit Cloud (or set up a self-hosted server), install the LiveKit CLI, and save your project's URL, API key, and API secret in a .env.local file.

Step 3: Install LiveKit Agents. Install the livekit-agents package, plus plugins for any AI providers you want to use.

Step 4: Write your agent. An agent is a Python class with instructions. An AgentSession connects it to the models that let it listen, think, and speak. A simplified version looks like this:

from livekit.agents import Agent, AgentSession, inference, TurnHandlingOptions

class Assistant(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are a friendly voice assistant. "
                         "Keep your answers short and clear."
        )

session = AgentSession(
    stt=inference.STT(model="deepgram/nova-3"),        # ears
    llm=inference.LLM(model="openai/chat-latest"),     # brain
    tts=inference.TTS(model="cartesia/sonic-3"),       # voice
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector()       # knows when to talk
    ),
)

In the full version, this code runs inside an entrypoint function that starts the session in a LiveKit room. LiveKit's Python starter project includes that boilerplate, so the easiest path is to start from it.

Step 5: Talk to it. Run the agent in development mode, then open LiveKit's Agent Console or Agents Playground in your browser and start talking.

Step 6: Improve the instructions. Most of the work after this point is writing clear instructions and testing real conversations, not writing more code.

LiveKit's APIs evolve quickly, so always check the official quickstart for the latest code before you build.

Giving your agent tools

A voice agent that can only chat is not very useful. The real value comes when it can take action or fetch real information. In LiveKit Agents, you do this with function tools.

A function tool is a normal Python function that you mark with the @function_tool decorator. The AI model reads the function's name and description, and calls it when the user asks for something it can help with.

from livekit.agents import function_tool

@function_tool
async def get_todays_sales() -> str:
    """Get today's total sales from the company dashboard."""
    # Call your own API here
    return "Today's sales are 42,300 dollars."

Now, when a user asks "How did sales go today?", the agent calls this function, gets the real number, and says it out loud. It does not have to guess.

Good function tools have three things in common:

  • A clear description. The AI picks tools based on their descriptions, so be specific.
  • Read-only when possible. If the agent only needs to look things up, do not give it tools that change or delete data.
  • Fast responses. Voice users notice delays. Slow jobs should run in the background while the agent keeps talking.

Taking it to production

Getting an agent to work on your laptop is the easy part. Making it reliable for real users takes a bit more planning.

Run it as an always-on service. In production, you start the agent in production mode and run it as a service on a server, so it restarts by itself after a crash or reboot. LiveKit Cloud can also host your agent for you.

Control who can join. Users connect to your agent through a LiveKit room. Your server should check who they are first, then give them a short-lived access token for a private room, so random people cannot use your agent.

Handle bad connections. Mobile users walk into elevators and lose signal. Plan for this by letting the agent wait for the user to reconnect, instead of ending the call the moment the connection drops.

Test conversations before every release. Small changes to instructions can change how an agent behaves. Automated conversation tests that simulate real users help you catch problems before your users do.

Connect phone calls if you need them. LiveKit supports telephony through SIP, so the same agent can answer phone calls, not just browser or app calls.

Keep secrets out of the code. Store API keys in environment variables, and sign any webhooks your agent sends, so other systems can trust the requests really came from your agent.

LiveKit vs hosted voice platforms

LiveKit is not the only way to build a voice agent. Hosted platforms like Retell AI and Vapi let you build one with much less code. Here is how they compare.

LiveKit Agents (Python)Hosted platforms (Retell AI, Vapi)
How you buildWrite the agent in Python codeMostly configure it in a dashboard
Time to first agentQuick to start, longer to polishUsually faster
ControlFull control over models, logic, and data flowControl within the platform's features
Custom tools and APIsAny Python code you can writeWebhooks and built-in integrations
HostingLiveKit Cloud or your own serversThe platform's cloud
Open sourceYesNo
Best forCustom agents with complex logic or strict data needsStandard receptionist and sales agents

Choose LiveKit if you have a developer, need custom logic or many tools, want to host everything yourself, or are building voice into your own product.

Choose a hosted platform if you want a standard AI receptionist or appointment setter live quickly, and you do not want to maintain code.

Both paths can produce excellent voice agents. The right choice depends on how custom your agent needs to be and who will maintain it.

Real example: Ava, our morning brief voice agent

We built Ava, a real-time voice agent using LiveKit Agents and Python, as a working prototype for a sample freight and warehousing company called Harbor Point Logistics. It shows how managers could skip checking several systems every morning and just ask for an update instead (project page). All the company data in the prototype is sample data.

Instead of opening dashboards, a manager opens a call page on their phone or laptop and simply asks Ava. She reads a short spoken brief on sales, delays, urgent tickets, and meetings, and answers follow-up questions in plain language. The speech-to-text, AI model, and voice all run through LiveKit Inference, so no extra accounts are needed.

Here is how it works:

  1. Secure sign-in. The user enters a team passcode once. The server checks it and gives them a short-lived LiveKit token for a private call room.
  2. Voice conversation. LiveKit sends Ava into the call. The user's speech is turned into text in real time, and a turn detector makes sure Ava does not cut them off mid-sentence.
  3. Function calling. The AI model decides which tool to use and fetches real data from a read-only company API, a knowledge base, and live weather and time. It never makes up numbers.
  4. Claude webhook. For deeper analysis, Ava sends a signed webhook to a Claude skill. It runs in the background while she keeps talking, and she reads the result when it arrives.
  5. Spoken answer and hang-up. Ava replies in a natural voice with a live transcript on screen. When the user is done, she says goodbye, ends the call, and saves it to the call history.
  6. Always-on deployment. The agent and call page run as services on an Ubuntu server with HTTPS, so they restart by themselves after a crash or reboot.

Here is what the build includes:

ResultWhat it means
11 function toolsCompany API, knowledge base, live weather, time, Claude analysis, and hang-up
Daily brief from 2 sourcesCompany data and live weather are fetched in parallel in a single tool call
20-second reconnect windowCalls survive network drops, because Ava waits for the caller to reconnect
10 automated conversation testsSimulations check Ava's behavior before every release

The prototype shows what this kind of agent can do for a real business: managers get the morning update hands-free, even while driving, and can ask follow-up questions without opening a single dashboard. Company data stays safe, because Ava can only read data, never change it, and every Claude request is signed.

The code for this project is available on GitHub.

Conclusion

LiveKit Agents makes it possible to build a real-time AI voice agent in Python that feels natural to talk to. It handles the hard parts of live audio, while you focus on what the agent should know and do.

The core idea is simple: the agent listens, thinks, and speaks, with turn detection keeping the conversation smooth. Function tools are what turn it from a chatbot into something genuinely useful, because they let it fetch real data and take real actions.

If you need a standard AI receptionist, a hosted platform may get you there faster. But if you need custom logic, many tools, full control over your data, or voice built into your own product, LiveKit and Python are a great fit.

Frequently asked questions

Is LiveKit free?

LiveKit and the LiveKit Agents framework are open source, so you can self-host them for free and only pay for your servers and AI models. LiveKit Cloud is a hosted option that runs the real-time infrastructure for you, and you can start with a free account.

Do I need to know Python to build a LiveKit voice agent?

Yes, for the Python version. LiveKit Agents also supports Node.js. LiveKit offers a no-code Agent Builder for quick prototypes in the browser, but it has fewer features than the full SDK.

Which AI models can I use with LiveKit?

Many. LiveKit supports a wide range of speech-to-text, LLM, and text-to-speech providers, plus realtime speech-to-speech models like OpenAI's realtime models. You can mix and match them.

Can a LiveKit agent answer phone calls?

Yes. LiveKit supports telephony through SIP, so the same agent can talk to callers on a regular phone line as well as in a browser or app.

Is LiveKit better than Retell AI or Vapi?

Not better, just different. LiveKit gives you more control and is open source, but needs more developer work. Retell AI and Vapi are faster to set up for standard use cases.

Book Consultation