Cookbook
Streaming Chatbot
Build a chat product with streaming output, message history, cancellation, and security boundaries.
A chat product needs more than one model call: it must manage history, forward incremental output, apply security filters, cancel requests, and control costs.
Recommended architecture
Web or app for display and cancellation → application backend for authentication, sessions, and rate limits → APIQIK for streaming generation.
Keep the API key only on the backend. Authenticate the user, load a trimmed conversation history, and forward APIQIK deltas to the frontend.
Core server logic
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["APIQIK_API_KEY"],
base_url="https://api.apiqik.com/v1",
)
def stream_answer(history, user_text):
messages = [
{
"role": "system",
"content": "You are a product support assistant. State uncertainty and do not invent features.",
},
*history[-12:],
{"role": "user", "content": user_text},
]
stream = client.chat.completions.create(
model="qwen-turbo",
messages=messages,
temperature=0.4,
max_tokens=800,
stream=True,
)
for chunk in stream:
if chunk.choices:
text = chunk.choices[0].delta.content
if text:
yield textForward yielded text through your own WebSocket, SSE, or chunked HTTP response. Never send an APIQIK key to the browser.
Conversation history
- Store structured messages rather than concatenating the conversation into one string.
- Limiting recent message count is a simple starting point; token-budget trimming is more robust.
- Compress older turns into factual summaries and retain recent turns verbatim.
- Extract key fields from large tool-result JSON rather than retaining it indefinitely.
- Set maximum turns, context size, and idle expiration for each session.
Product experience checklist
| Feature | Implementation focus |
|---|---|
| Typewriter effect | Batch small chunks to reduce rendering jitter. |
| Stop generation | Close the APIQIK upstream connection when the frontend cancels. |
| Regenerate | Reuse history before the current turn without adding the failed reply again. |
| Source citations | Use stable RAG source IDs; the application renders links. |
| Failure messages | Distinguish balance, rate-limit, timeout, and system failures with actionable guidance. |