On this tutorial, we discover construct an Agentic Voice AI Assistant able to understanding, reasoning, and responding by way of pure speech in actual time. We start by establishing a self-contained voice intelligence pipeline that integrates speech recognition, intent detection, multi-step reasoning, and text-to-speech synthesis. Alongside the way in which, we design an agent that listens to instructions, identifies objectives, plans applicable actions, and delivers spoken responses utilizing fashions similar to Whisper and SpeechT5. We strategy your entire system from a sensible standpoint, demonstrating how notion, reasoning, and execution work together seamlessly to create an autonomous conversational expertise. Take a look at the FULL CODES here.
import subprocess
import sys
import json
import re
from datetime import datetime
from typing import Dict, Record, Tuple, Any
def install_packages():
packages = ['transformers', 'torch', 'torchaudio', 'datasets', 'soundfile',
'librosa', 'IPython', 'numpy']
for pkg in packages:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '-q', pkg])
print("🤖 Initializing Agentic Voice AI...")
install_packages()
import torch
import soundfile as sf
import numpy as np
from transformers import (AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline,
SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan)
from IPython.show import Audio, show, HTML
import warnings
warnings.filterwarnings('ignore')We start by putting in all of the important libraries, together with Transformers, Torch, and SoundFile, to allow speech recognition and synthesis. We additionally configure the atmosphere to suppress warnings and guarantee clean execution all through the voice AI setup. Take a look at the FULL CODES here.
class VoiceAgent:
def __init__(self):
self.reminiscence = []
self.context = {}
self.instruments = {}
self.objectives = []
def understand(self, audio_input: str) -> Dict[str, Any]:
intent = self._extract_intent(audio_input)
entities = self._extract_entities(audio_input)
sentiment = self._analyze_sentiment(audio_input)
notion = {
'textual content': audio_input,
'intent': intent,
'entities': entities,
'sentiment': sentiment,
'timestamp': datetime.now().isoformat()
}
self.reminiscence.append(notion)
return notion
def _extract_intent(self, textual content: str) -> str:
text_lower = textual content.decrease()
intent_patterns = {
'create': ['create', 'make', 'generate', 'write'],
'search': ['search', 'find', 'look for', 'show me'],
'analyze': ['analyze', 'explain', 'understand', 'what is'],
'calculate': ['calculate', 'compute', 'how much', 'sum'],
'schedule': ['schedule', 'plan', 'set reminder', 'meeting'],
'translate': ['translate', 'say in', 'convert to'],
'summarize': ['summarize', 'brief', 'tldr', 'overview']
}
for intent, key phrases in intent_patterns.objects():
if any(kw in text_lower for kw in key phrases):
return intent
return 'dialog'
def _extract_entities(self, textual content: str) -> Dict[str, List[str]]:
entities = {
'numbers': re.findall(r'd+', textual content),
'dates': re.findall(r'bd{1,2}/d{1,2}/d{2,4}b', textual content),
'occasions': re.findall(r'bd{1,2}:d{2}s*(?:am|pm)?b', textual content.decrease()),
'emails': re.findall(r'b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Z|a-z]{2,}b', textual content)
}
return {ok: v for ok, v in entities.objects() if v}
def _analyze_sentiment(self, textual content: str) -> str:
optimistic = ['good', 'great', 'excellent', 'happy', 'love', 'thank']
adverse = ['bad', 'terrible', 'sad', 'hate', 'angry', 'problem']
text_lower = textual content.decrease()
pos_count = sum(1 for phrase in optimistic if phrase in text_lower)
neg_count = sum(1 for phrase in adverse if phrase in text_lower)
if pos_count > neg_count:
return 'optimistic'
elif neg_count > pos_count:
return 'adverse'
return 'impartial'Right here, we implement the notion layer of our agent. We design strategies to extract intents, entities, and sentiment from spoken textual content, enabling the system to grasp consumer enter inside its context. Take a look at the FULL CODES here.
def cause(self, notion: Dict) -> Dict[str, Any]:
intent = notion['intent']
reasoning = {
'purpose': self._identify_goal(intent),
'stipulations': self._check_prerequisites(intent),
'plan': self._create_plan(intent, notion['entities']),
'confidence': self._calculate_confidence(notion)
}
return reasoning
def act(self, reasoning: Dict) -> str:
plan = reasoning['plan']
outcomes = []
for step in plan['steps']:
end result = self._execute_step(step)
outcomes.append(end result)
response = self._generate_response(outcomes, reasoning)
return response
def _identify_goal(self, intent: str) -> str:
goal_mapping = {
'create': 'Generate new content material',
'search': 'Retrieve info',
'analyze': 'Perceive and clarify',
'calculate': 'Carry out computation',
'schedule': 'Arrange time-based duties',
'translate': 'Convert between languages',
'summarize': 'Condense info'
}
return goal_mapping.get(intent, 'Help consumer')
def _check_prerequisites(self, intent: str) -> Record[str]:
prereqs = {
'search': ['internet access', 'search tool'],
'calculate': ['math processor'],
'translate': ['translation model'],
'schedule': ['calendar access']
}
return prereqs.get(intent, ['language understanding'])
def _create_plan(self, intent: str, entities: Dict) -> Dict:
plans = {
'create': {'steps': ['understand_requirements', 'generate_content', 'validate_output'], 'estimated_time': '10s'},
'analyze': {'steps': ['parse_input', 'analyze_components', 'synthesize_explanation'], 'estimated_time': '5s'},
'calculate': {'steps': ['extract_numbers', 'determine_operation', 'compute_result'], 'estimated_time': '2s'}
}
default_plan = {'steps': ['understand_query', 'process_information', 'formulate_response'], 'estimated_time': '3s'}
return plans.get(intent, default_plan)We now give attention to reasoning and planning. We educate the agent establish objectives, verify stipulations, and generate structured multi-step plans to execute consumer instructions logically. Take a look at the FULL CODES here.
def _calculate_confidence(self, notion: Dict) -> float:
base_confidence = 0.7
if notion['entities']:
base_confidence += 0.15
if notion['sentiment'] != 'impartial':
base_confidence += 0.1
if len(notion['text'].cut up()) > 5:
base_confidence += 0.05
return min(base_confidence, 1.0)
def _execute_step(self, step: str) -> Dict:
return {'step': step, 'standing': 'accomplished', 'output': f'Executed {step}'}
def _generate_response(self, outcomes: Record, reasoning: Dict) -> str:
intent = reasoning['goal']
confidence = reasoning['confidence']
prefix = "I perceive you wish to" if confidence > 0.8 else "I feel you are asking me to"
response = f"{prefix} {intent.decrease()}. "
if len(self.reminiscence) > 1:
response += "Based mostly on our dialog, "
response += f"I've analyzed your request and accomplished {len(outcomes)} steps. "
return responseOn this part, we implement helper capabilities that calculate confidence ranges, execute every deliberate step, and generate significant pure language responses for the consumer. Take a look at the FULL CODES here.
class VoiceIO:
def __init__(self):
print("Loading voice fashions...")
machine = "cuda:0" if torch.cuda.is_available() else "cpu"
self.stt_pipe = pipeline("automatic-speech-recognition", mannequin="openai/whisper-base", machine=machine)
self.tts_processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts")
self.tts_model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts")
self.vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")
self.speaker_embeddings = torch.randn(1, 512) * 0.1
print("✓ Voice I/O prepared")
def pay attention(self, audio_path: str) -> str:
end result = self.stt_pipe(audio_path)
return end result['text']
def converse(self, textual content: str, output_path: str = "response.wav") -> Tuple[str, np.ndarray]:
inputs = self.tts_processor(textual content=textual content, return_tensors="pt")
speech = self.tts_model.generate_speech(inputs["input_ids"], self.speaker_embeddings, vocoder=self.vocoder)
sf.write(output_path, speech.numpy(), samplerate=16000)
return output_path, speech.numpy()
class AgenticVoiceAssistant:
def __init__(self):
self.agent = VoiceAgent()
self.voice_io = VoiceIO()
self.interaction_count = 0
def process_voice_input(self, audio_path: str) -> Dict:
text_input = self.voice_io.pay attention(audio_path)
notion = self.agent.understand(text_input)
reasoning = self.agent.cause(notion)
response_text = self.agent.act(reasoning)
audio_path, audio_array = self.voice_io.converse(response_text)
self.interaction_count += 1
return {
'input_text': text_input,
'notion': notion,
'reasoning': reasoning,
'response_text': response_text,
'audio_path': audio_path,
'audio_array': audio_array
}We arrange the core voice enter and output pipeline utilizing Whisper for transcription and SpeechT5 for speech synthesis. We then combine these with the agent’s reasoning engine to kind an entire interactive assistant. Take a look at the FULL CODES here.
def display_reasoning(self, end result: Dict):
html = f"""
🤖 Agent Reasoning Course of
📥 INPUT: {end result['input_text']}
🧠 PERCEPTION:
- Intent: {end result['perception']['intent']}
- Entities: {end result['perception']['entities']}
- Sentiment: {end result['perception']['sentiment']}
💭 REASONING:
- Aim: {end result['reasoning']['goal']}
- Plan: {len(end result['reasoning']['plan']['steps'])} steps
- Confidence: {end result['reasoning']['confidence']:.2%}
💬 RESPONSE: {end result['response_text']}
"""
show(HTML(html))
def run_agentic_demo():
print("n" + "="*70)
print("🤖 AGENTIC VOICE AI ASSISTANT")
print("="*70 + "n")
assistant = AgenticVoiceAssistant()
situations = [
"Create a summary of machine learning concepts",
"Calculate the sum of twenty five and thirty seven",
"Analyze the benefits of renewable energy"
]
for i, scenario_text in enumerate(situations, 1):
print(f"n--- State of affairs {i} ---")
print(f"Simulated Enter: '{scenario_text}'")
audio_path, _ = assistant.voice_io.converse(scenario_text, f"input_{i}.wav")
end result = assistant.process_voice_input(audio_path)
assistant.display_reasoning(end result)
print("n🔊 Enjoying agent's voice response...")
show(Audio(end result['audio_array'], charge=16000))
print("n" + "-"*70)
print(f"n✅ Accomplished {assistant.interaction_count} agentic interactions")
print("n🎯 Key Agentic Capabilities Demonstrated:")
print(" • Autonomous notion and understanding")
print(" • Intent recognition and entity extraction")
print(" • Multi-step reasoning and planning")
print(" • Aim-driven motion execution")
print(" • Pure language response technology")
print(" • Reminiscence and context administration")
if __name__ == "__main__":
run_agentic_demo()Lastly, we run a demo to visualise the agent’s full reasoning course of and listen to it reply. We check a number of situations to showcase notion, reasoning, and voice response working in excellent concord.
In conclusion, we constructed an clever voice assistant that understands what we are saying and likewise causes, plans, and speaks like a real agent. We skilled how notion, reasoning, and motion work in concord to create a pure and adaptive voice interface. By means of this implementation, we purpose to bridge the hole between passive voice instructions and autonomous decision-making, demonstrating how agentic intelligence can improve human–AI voice interactions.
Take a look at the FULL CODES here. Be happy to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be at liberty to comply with us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
