Skip to main content

AI Voice Agents

Build AI-powered voice agents that handle conversations autonomously, understand context, and can take actions through function calling.

Overview

AI agents in Kallglot can:
  • Understand natural language in 25+ languages
  • Respond with natural-sounding speech
  • Follow custom instructions via system prompts
  • Access knowledge bases for contextual answers
  • Call external functions/tools to take actions
  • Escalate to human agents when needed

Creating an AI Agent Session

Kallglot does not currently provide official SDKs. The helper functions in this guide, such as createKallglotSession(...), are placeholders for direct HTTP requests to the API.
const session = await createKallglotSession({
  mode: 'ai_agent',
  source_language: 'en',
  target_language: 'en',

  ai_agent: {
    // Custom system prompt
    system_prompt: `You are a helpful customer service agent for Acme Corp.
You assist customers with order inquiries, returns, and general questions.
Be friendly, professional, and concise.
If you cannot help with something, offer to transfer to a human agent.`,

    // Voice selection
    voice: 'alloy',

    // Temperature for response generation
    temperature: 0.7,

    // Knowledge base for context
    knowledge_base_id: 'kb_01ABC123',

    // Function calling
    tools: [
      {
        name: 'lookup_order',
        description: 'Look up an order by order number or customer email',
        parameters: {
          type: 'object',
          properties: {
            order_number: {
              type: 'string',
              description: 'The order number (e.g., ORD-12345)'
            },
            email: {
              type: 'string',
              description: 'Customer email address'
            }
          }
        }
      },
      {
        name: 'initiate_return',
        description: 'Start a return process for an order',
        parameters: {
          type: 'object',
          properties: {
            order_number: { type: 'string' },
            reason: { type: 'string' },
            items: {
              type: 'array',
              items: { type: 'string' }
            }
          },
          required: ['order_number', 'reason']
        }
      },
      {
        name: 'transfer_to_human',
        description: 'Transfer the call to a human agent',
        parameters: {
          type: 'object',
          properties: {
            reason: { type: 'string' },
            department: {
              type: 'string',
              enum: ['support', 'sales', 'billing']
            }
          }
        }
      }
    ]
  }
});

System Prompts

The system prompt defines your agent’s personality, capabilities, and behavior.

Best Practices

You are [Role] for [Company].

## Your Capabilities
- [Capability 1]
- [Capability 2]

## Guidelines
- Be [tone] and [style]
- [Specific instruction]
- [Boundary]

## Escalation
Transfer to a human agent when:
- [Condition 1]
- [Condition 2]

Example: E-commerce Support Agent

You are a customer support agent for TechGadgets, an electronics retailer.

## Your Capabilities
- Look up order status and tracking information
- Process returns and exchanges for orders within 30 days
- Answer questions about products and warranties
- Help with account issues

## Guidelines
- Be friendly and empathetic
- Keep responses concise (2-3 sentences when possible)
- Always verify the customer's identity before discussing order details
- Never share personal information from other customers

## Escalation
Transfer to a human agent when:
- Customer requests to speak with a person
- Issue involves a refund over $500
- Customer is upset or frustrated after 2 failed resolution attempts
- Technical issues you cannot resolve

Handling Function Calls

When the AI agent needs to call a function, you’ll receive a webhook:
app.post('/webhooks/kallglot/tool_call', async (req, res) => {
  const { session_id, tool } = req.body.data;

  let result;

  switch (tool.name) {
    case 'lookup_order':
      result = await lookupOrder(tool.arguments);
      break;

    case 'initiate_return':
      result = await initiateReturn(tool.arguments);
      break;

    case 'transfer_to_human':
      result = await transferToHuman(session_id, tool.arguments);
      break;

    default:
      result = { error: 'Unknown function' };
  }

  // Send the result back to the agent
  await submitKallglotToolResult(session_id, {
    tool_call_id: tool.call_id,
    result: JSON.stringify(result)
  });

  res.status(200).send('OK');
});

async function lookupOrder({ order_number, email }) {
  const order = await db.orders.findOne({
    $or: [
      { order_number },
      { customer_email: email }
    ]
  });

  if (!order) {
    return {
      found: false,
      message: 'No order found with that information'
    };
  }

  return {
    found: true,
    order_number: order.order_number,
    status: order.status,
    items: order.items,
    tracking_number: order.tracking_number,
    estimated_delivery: order.estimated_delivery
  };
}

Knowledge Bases

Connect a knowledge base to give your agent access to company information:

Creating a Knowledge Base

const kb = await createKnowledgeBase({
  name: 'Product Documentation',
  description: 'Product specs, FAQs, and troubleshooting guides'
});

// Add documents
await addKnowledgeBaseDocuments(kb.id, [
  {
    type: 'url',
    url: 'https://your-site.com/products'
  },
  {
    type: 'file',
    file: fs.createReadStream('product-manual.pdf'),
    filename: 'product-manual.pdf'
  },
  {
    type: 'text',
    content: 'FAQ: How to reset your device...',
    metadata: { category: 'troubleshooting' }
  }
]);

Using the Knowledge Base

const session = await createKallglotSession({
  mode: 'ai_agent',
  ai_agent: {
    knowledge_base_id: kb.id,
    // Agent will automatically search the KB when relevant
  }
});

Voice Configuration

Available Voices

Voice IDStyleBest For
alloyNeutral, clearGeneral purpose
echoWarm, friendlyCustomer service
fableExpressiveStorytelling, engagement
onyxDeep, authoritativeProfessional, formal
novaEnergetic, upbeatSales, marketing
shimmerCalm, soothingSupport, helpdesk

Voice Parameters

ai_agent: {
  voice: 'echo',
  voice_settings: {
    speed: 1.0,        // 0.5 - 2.0
    pitch: 0,          // -12 to 12 semitones
    stability: 0.75,   // 0 - 1 (higher = more consistent)
    similarity: 0.85   // 0 - 1 (higher = more similar to original voice)
  }
}

Multilingual Agents

AI agents can handle conversations in multiple languages:
const session = await createKallglotSession({
  mode: 'ai_agent',
  source_language: 'auto',  // Detect customer's language
  target_language: 'auto',  // Respond in same language

  ai_agent: {
    system_prompt: `You are a multilingual support agent.
Respond in the same language the customer uses.
You are fluent in English, German, French, and Spanish.`,

    // Voice that works well for multiple languages
    voice: 'alloy',

    multilingual: {
      enabled: true,
      supported_languages: ['en', 'de', 'fr', 'es'],
      fallback_language: 'en'
    }
  }
});

Escalation to Human Agents

Handle handoffs gracefully:
async function transferToHuman(sessionId, { reason, department }) {
  // 1. Get session summary
  const session = await retrieveKallglotSession(sessionId);
  const transcript = await getKallglotTranscript(sessionId);

  // 2. Find available agent
  const agent = await findAvailableAgent(department);

  if (!agent) {
    return {
      success: false,
      message: 'No agents available. Please try again later.'
    };
  }

  // 3. Create a new session for the human agent
  const humanSession = await createKallglotSession({
    mode: 'bidirectional_translation',
    source_language: session.source_language,
    target_language: agent.language
  });

  // 4. Transfer the call
  await transferKallglotSession(sessionId, {
    target_session_id: humanSession.id,
    context: {
      summary: transcript.summary,
      sentiment: transcript.sentiment,
      reason_for_transfer: reason
    }
  });

  return {
    success: true,
    message: 'Transferring you to an agent now.'
  };
}

Conversation Context

Access conversation context during the session:
// Get current conversation state
const context = await getKallglotSessionContext(sessionId);

console.log(context);
// {
//   turn_count: 5,
//   customer_sentiment: 'positive',
//   topics_discussed: ['order_status', 'shipping'],
//   entities: {
//     order_number: 'ORD-12345',
//     product: 'Wireless Headphones'
//   },
//   intent_history: [
//     { intent: 'check_order', confidence: 0.95 },
//     { intent: 'ask_shipping', confidence: 0.88 }
//   ]
// }

Error Handling

Handle agent errors gracefully:
session.on('error', async (error) => {
  console.error('Agent error:', error);

  if (error.code === 'llm_timeout') {
    // LLM took too long - provide fallback response
    await session.speak("I'm sorry, I'm having trouble processing that. Let me transfer you to a colleague.");
    await transferToHuman(session.id, { reason: 'technical_issue' });
  }
});

session.on('no_match', async (data) => {
  // Agent couldn't understand the customer
  if (data.consecutive_failures >= 3) {
    await session.speak("I'm having trouble understanding. Would you like to speak with a person?");
  }
});

Analytics

Track agent performance:
// After session ends
const analysis = await requestKallglotAnalysis(sessionId, {
  analyses: ['quality', 'sentiment', 'summary']
});

// Log metrics
await analytics.track('ai_agent_session', {
  session_id: sessionId,
  duration: session.duration,
  turns: context.turn_count,
  resolution: analysis.results.summary.outcome,
  sentiment: analysis.results.sentiment.label,
  escalated: session.end_reason === 'transfer',
  quality_score: analysis.results.quality.score
});

Best Practices

Avoid overly long system prompts. Focus on the most important behaviors and let the agent handle edge cases naturally.
Test your agent with actual customer conversations, not just scripted tests. Real conversations are more varied.
Regularly review transcripts and analytics. Update your system prompt and tools based on actual conversations.
Always provide a way for customers to reach a human. Make the escalation trigger easy for the agent to understand.
If the customer is silent, have the agent prompt them gently rather than repeating or hanging up.