Category: Anything

Today started with a very reasonable engineering plan:

Run Qwen/Qwen3.6-27B-FP8 on SGLang.

Simple.

Clean.

Responsible.

Naturally, within minutes, I had rented the wrong GPU.

Act 1: The A40 Incident

I started with an A40.

On paper, this made sense if you squint hard enough. It has 48 GB of VRAM. It is available. It is not outrageously expensive. It looks like the kind of GPU you rent when you want to “just test something quickly.”

Unfortunately, the thing I wanted to test was FP8.

The A40 is Ampere.

Ampere is not where this story belongs.

So the first lesson arrived immediately:

VRAM is not a personality.
Architecture matters.

The A40 was terminated.

No ceremony. No farewell message. Just goodbye.

Act 2: Fine, We’re Doing Blackwell

After briefly pretending I was going to be practical, I rented the machine I should have rented in the first place:

GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
RAM: 140 GB
vCPU: 16
Persistent disk: 120 GB
Container disk: 80 GB

Now we were speaking the right language.

This was no longer “can I squeeze this model into a GPU and pray?”

This was:

Let’s see if a single Blackwell card can become a real inference lane.

The target stack:

Model:   Qwen/Qwen3.6-27B-FP8
Runtime: SGLang
GPU:     RTX PRO 6000 Blackwell
API:     OpenAI-compatible

Beautiful.

Then CUDA showed up with a chair.

Act 3: CUDA Userspace Betrayal

The first container image I used was:

runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404

It looked fine.

PyTorch saw the GPU:

NVIDIA RTX PRO 6000 Blackwell Server Edition

The driver reported CUDA 13 support.

Everything seemed okay.

Then SGLang said:

Failed to get device capability: SM 12.x requires CUDA >= 12.9.

And there it was.

The machine had a CUDA 13-capable driver, but the container userspace/toolkit was CUDA 12.8.

That is the kind of error that makes you stare at the terminal like it owes you money.

The fix was to stop being clever and use the official SGLang CUDA 13 image:

lmsysorg/sglang:dev-cu13

That was the correct move.

The container initially needed a boring foreground command to stay alive, so I used:

sleep infinity

Not glamorous.

Extremely effective.

Act 4: The Boring Launch That Proved the Stack

Inside the working container, I set the Hugging Face cache to the persistent volume:

export HF_HOME=/workspace/hf-cache
export HF_XET_HIGH_PERFORMANCE=1

Then I launched the model with the most basic command possible:

python -m sglang.launch_server \
    --model-path Qwen/Qwen3.6-27B-FP8 \
    --host 0.0.0.0 \
    --port 30000 \
    --trust-remote-code

This was not the final performance config.

This was the “does the thing even boot?” config.

And it did.

The logs said exactly what I wanted to see:

Detected fp8 checkpoint.
Load weight begin. avail mem=94.27 GB
Auto-detected template features:
  reasoning_parser=qwen3
  tool_call_parser=qwen3_coder
Uvicorn running on http://0.0.0.0:30000

At this point, the model was alive.

But alive is not the same thing as fast.

That came next.

Act 5: The Actual Config That Made It Fly

The basic launch proved compatibility.

The tuned config made it stupid.

This was the real serving command:

sglang serve \
  --model-path Qwen/Qwen3.6-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mamba-scheduler-strategy extra_buffer \
  --page-size 64 \
  --chunked-prefill-size 16384 \
  --max-prefill-tokens 32768 \
  --mem-fraction-static 0.75 \
  --enable-cache-report

This is the important distinction.

The default launch got Qwen running.

This config made it feel production-shaped.

The explicit Qwen parsers aligned SGLang with the model’s reasoning and tool-call format. EAGLE speculative decoding helped the decode path. Chunked prefill and the scheduler settings mattered because the workload was not a cute 512-token prompt. It was a large, repeated, agent-shaped prompt.

This is where the experiment stopped feeling like:

“Cool, the model runs.”

And started feeling like:

“Wait. This might actually serve agents.”

Act 6: The First Real Benchmark

The benchmark prompt was intentionally obnoxious.

I repeated a small reasoning problem 400 times to create a roughly 20k-token prompt:

python tools/sglang_bench.py \
  --runs 1 \
  --prompt-repeat 400 \
  --max-tokens 8000

The shape of the result:

Prompt tokens:      ~19,609
Completion tokens:  ~2,300–2,500
TTFT:               ~1.3–2.5s
Decode throughput:  ~79–82 tok/s
End-to-end:         ~30–33s

That was the first “oh no, this is real” moment.

A 20k-token prompt.

Thinking enabled.

First token in around 1–2.5 seconds.

Sustained decode around 80 tokens/sec.

That is not a toy benchmark anymore. That is very close to the shape of a real agent request.

Act 7: Five Terminals, Then Thirteen, Then Bad Decisions

At first I ran one request.

Then several.

Then five terminals at once.

It still held.

Then thirteen concurrent requests.

Still held.

The SGLang logs were the fun part:

cuda graph: True
accept len: ~3
accept rate: ~0.7+
gen throughput: ~750–830 tok/s aggregate

That is where the test stopped being about Qwen alone.

This became a scheduler test.

SGLang was doing real work here: batching, scheduling, speculative decoding, and keeping the GPU busy without the whole system turning into soup.

Act 8: The 50-User Hunger Games

At some point, incremental testing lost the argument.

So I ran:

python tools/sglang_bench.py \
  --users 50 \
  --requests-per-user 1 \
  --prompt-repeat 400 \
  --max-tokens 8000 \
  --timeout 600

This fired 50 long-context requests at the server.

Each request had roughly:

~19.6k prompt tokens
~2k–3k completion tokens
thinking enabled
streaming enabled

The result:

successful_requests=50
failed_requests=0
wall_time=206.105s

Aggregate Tokens
  prompt=980450
  completion=120586
  total=1101036

Aggregate Throughput
  prompt_per_wall=4757.04 tok/s
  completion_per_wall=585.07 tok/s
  total_per_wall=5342.11 tok/s

Per-Request Distribution
  ttft_sec: min=1.311 median=1.905 p90=3.268 max=10.944
  e2e_sec: min=28.256 median=42.128 p90=52.550 max=59.781
  completion_decode_tps: min=47.512 median=60.833 p90=64.546 max=78.717

The headline:

50 requests.
0 failures.

The median TTFT stayed under 2 seconds.

The median decode throughput was around 60 tokens/sec.

One request had a TTFT outlier around 11 seconds, which is exactly the kind of thing a real load test should reveal.

But the system did not collapse.

No dramatic death spiral.

No wall of timeouts.

No “please reduce your expectations” message from the GPU.

Just throughput.

Act 9: Important Honesty Break

The 50-user test was useful, but I would not oversell it as a perfect 50-simultaneous-client benchmark yet.

The benchmark script uses blocking urllib calls wrapped in asyncio.to_thread().

That means it works well as a practical chaos/load test, but Python thread scheduling can still affect how perfectly simultaneous the requests are.

Before turning this into a formal benchmark claim, I want a true async streaming client using something like httpx.AsyncClient.

The next harness should record:

virtual user scheduled time
actual request send time
first byte
first token
end of stream
usage tokens
server queue behavior

Still, the result is extremely useful.

It showed that this stack can handle a nasty burst of large-context, thinking-enabled requests without falling apart.

Act 10: The 1000-User Crime Scene

Then I did the unreasonable thing.

python tools/sglang_bench.py \
  --users 1000 \
  --requests-per-user 1 \
  --prompt-repeat 400 \
  --max-tokens 8000 \
  --timeout 600

This was not a benchmark.

This was violence.

A thousand requests.

Each with around 20k prompt tokens.

That is about 19.6 million prompt tokens thrown at one GPU.

SGLang did not politely ask me to stop. It simply showed me the wall:

#running-req: 13
#queue-req: 984
#pending-token: 19285959

There it was.

For this workload, the active running set was around 13 requests.

The rest queued.

That is actually a great result, because now I know the shape of the bottleneck.

The server did not mysteriously explode. It exposed its scheduling behavior.

But the most important line was this:

#cached-token: 19584
#new-token: 64

That line is the entire production story.

The prompt was mostly repeated across requests. SGLang reused the prefix. It did not recompute the entire ~19.6k-token prompt from scratch every time.

That is not a benchmark trick.

That is exactly what real agent traffic often looks like:

same system prompt
same product instructions
same tool descriptions
same safety rules
same app context
different user request

Prefix caching is not just a performance optimization.

For agent systems, prefix caching can be the difference between “this is impossible” and “this is a production lane.”

Act 11: Can This Replace GLM 4.7?

Not proven yet.

This test proves the serving side is real.

It does not prove the model quality is good enough.

For my use case, replacing GLM 4.7 is not about whether Qwen can answer a math prompt. It has to survive actual agent work:

tool-call correctness
JSON/schema validity
long-context instruction retention
repair behavior after failed tools
multi-step planning
page/code generation quality
cost per successful task

The scary failure mode is not obvious stupidity.

The scary failure mode is almost-good-enough behavior.

A model can look fast, sound smart, and still make slightly more tool mistakes. In agent systems, a tiny quality regression can erase an infrastructure win.

So the next real test is:

GLM 4.7 production baseline
vs
Qwen3.6-27B-FP8 on SGLang

Same harness.

Same prompts.

Same tool traces.

Same scoring.

Same production tasks.

Only then can I say whether it replaces GLM 4.7.

What I Actually Learned

1. Do not rent architecture by VRAM alone

The A40 had VRAM.

It was still the wrong GPU.

For official FP8 inference, architecture matters.

2. Blackwell wants the right CUDA userspace

The driver can be fine. PyTorch can see the GPU. SGLang can still complain if the container userspace is wrong.

Use the right image.

In this case:

lmsysorg/sglang:dev-cu13

3. The boring launch is only the beginning

The basic launch proved the model could run.

The tuned config produced the real performance.

Production inference is not just:

load model
pray

It is:

model format
runtime
scheduler
speculative decoding
cache behavior
memory layout
workload shape

All of those have to line up.

4. SGLang’s prefix cache is a big deal

The 1000-user chaos test showed:

#cached-token: 19584
#new-token: 64

That is the line I care about most.

Agent workloads repeat huge prefixes constantly. If the runtime can exploit that, the economics change.

5. One RTX PRO 6000 Blackwell is not a toy

At around $1,500/month, this is not “cheap” in the casual sense.

But if it replaces enough API spend, gives predictable latency, avoids rate limits, and supports production agent workloads, it becomes a serious infrastructure option.

The question is not:

Is $1,500 cheap?

The question is:

How many successful agent tasks per month can this box serve compared to managed API spend?

That is the real math.

Final Take

This started as a bad A40 rental.

It ended with:

Qwen/Qwen3.6-27B-FP8
SGLang
RTX PRO 6000 Blackwell
CUDA 13
EAGLE speculative decoding
prefix caching
50-request chaos test
0 failures

The serving side is no longer theoretical.

The model still has to prove itself against GLM 4.7 on real agent tasks.

But as an inference stack?

This thing is alive.

And it is way more serious than I expected when I typed the first command.

A Guide to State, Streams, and Safety in Python

---

Chapter 1: The Foundation (async & await)

Before we build complex structures, we must understand the bricks.

The Problem: Blocking

In standard Python (Synchronous), the computer is single-minded. If you ask it to download a file, it stops everything else. The entire universe freezes until that file is finished.

The Solution: Asynchrony

Asynchronous Python allows the computer to multitask while waiting.

• async def: This defines a Coroutine. It tells Python, “This function can be paused.” It doesn’t run immediately; it returns a “pending task.”
• await: This is the Pause Button. It tells Python, “I need this result to continue, but while I wait, you can go do other work.”

The Rule: You can only use await inside an async def function.

---

Chapter 2: The Guard (Context Managers)

Context Managers are about Safety. They guarantee that resources (files, sockets, connections) are cleaned up, no matter what happens.

1. The “Sandwich” Model

Think of a context manager as a sandwich.

• Top Bread (Setup): Open the door.
• The Filling (Body): Do the work.
• Bottom Bread (Teardown): Close the door.

2. The Shift to Async

Standard context managers (with) are synchronous. If the “Top Bread” involves connecting to a slow database, your program freezes.

We need Async Context Managers (async with) to allow pausing during the Setup and Teardown phases.

3. The Magic Methods

To build one, you need a class with two specific methods.

• async def __aenter__(self):
  • Role: The Entry.
  • Why Async? So you can await a connection setup (e.g., await db.connect()).
  • Return: Whatever you return here becomes the variable after as.
• async def __aexit__(self, exc_type, ...):
  • Role: The Exit.
  • Why Async? So you can await a graceful shutdown (e.g., await db.disconnect()).
  • Guarantee: This runs even if the code crashes.

---

Chapter 3: The Stream (Generators)

Generators are about Flow. They allow you to process data one piece at a time, rather than waiting for the whole batch.

1. return vs yield

• return: “Here is the finished box. I am done. Goodbye.” (Function ends).
• yield: “Here is one item. I am pausing. Come back when you need the next one.” (Function stays alive).

2. Async Generators

An Async Generator combines yield with await.

• It can pause to fetch data (using await).
• It can deliver that data to you immediately (using yield).

Usage: You consume an async generator using async for, which repeatedly asks, “Do you have another item?” until the generator is empty.

---

Chapter 4: The Grand Unification (The Pattern)

This is the architecture used by professional AI SDKs (like Strands, OpenAI, Anthropic). It combines Safety (Context Manager) with Streaming (Generator).

The Code Reference

Python

import asyncio

# --- COMPONENT 1: THE CONTEXT MANAGER (The Safety Wrapper) ---
class SetupSession:
    
    # 1. INIT: Runs instantly. Sets up internal state.
    def __init__(self):
        print("[Init] Creating Session Object...")
        self._session_id = "USER_123"
    
    # Helper method we can call later
    def get_id(self):
        return self._session_id

    # 2. ENTRY (__aenter__): The "Top Bread".
    # We use 'async' here so we can await slow things (like Auth).
    async def __aenter__(self):
        print("[Enter] Connecting to Server...")
        await asyncio.sleep(0.1)  # Simulate network lag
        return self  # Returns 'self' so we can use methods like .get_id()

    # 3. EXIT (__aexit__): The "Bottom Bread".
    # Guaranteed to run. We use 'async' to close gracefully.
    async def __aexit__(self, exc_type, exc_val, exc_tb):
        print("[Exit] Closing Connection...")
        await asyncio.sleep(0.1)


# --- COMPONENT 2: THE GENERATOR (The Data Stream) ---
async def run_agent():
    print("   [Stream] Agent Thinking...")
    
    # Yield 1: Deliver data, then pause function.
    yield "Hello"   
    
    # Await: Go do other work while we wait for the next thought.
    await asyncio.sleep(0.1) 
    
    # Yield 2: Deliver next piece of data.
    yield "Human"   
    yield "!"       


# --- COMPONENT 3: EXECUTION (The Client) ---
async def main():
    # STEP A: 'async with' invokes __aenter__
    # The session is now "Open" and safe.
    async with SetupSession() as session:
        print(f"   [Logic] Session Active: {session.get_id()}")
        
        # STEP B: 'async for' pulls data from the generator
        # The session remains open while we stream.
        async for token in run_agent():
            print(f"   [Received]: {token}")

    # STEP C: Block ends. 'async with' invokes __aexit__ automatically.

if __name__ == "__main__":
    asyncio.run(main())

---

Appendix: The “Why” Cheat Sheet

Concept	The Question it Answers	The Keyword
Async/Await	“How do I wait for IO without freezing?”	await
Context Manager	“How do I ensure this closes safely?”	async with
Generator	“How do I get data piece-by-piece?”	yield
Async Iterator	“How do I consume a generator?”	async for
__aenter__	“What happens when the block starts?”	(Magic Method)
__aexit__	“What happens when the block ends?”	(Magic Method)

---

We’ve all been there. You build a RAG (Retrieval-Augmented Generation) pipeline, dump your documents into a vector database, and feel like a wizard. You search for ‘guidelines for remote work’ and it finds documents about ‘working from home policy.’ It feels like magic.

Then, you put it into production.

A user searches for a specific error code like ERR-502 or a specific product SKU like XJ-900.

The result? Hallucinations, or worse, totally irrelevant documents that just happen to share a similar ‘vibe’ in the embedding space.

The hard truth is that Vector Search is not a silver bullet. It understands meaning, but it is terrible at precision.

The Two Extremes of Search

To understand why your AI agent is failing, you have to look at the two distinct ways computers search for information.

1. The ‘Vibe’ Search (Vector Embeddings)

Vector search turns text into numbers (embeddings). It finds concepts that are semantically close.

Superpower: Understanding synonyms and intent (e.g., ‘monitor’ = ‘display’).

Kryptonite: Exact matches. To a vector model, ‘Version 1.2’ and ‘Version 2.1’ might look nearly identical because they are semantically similar concepts, even though they are completely different technical facts.

2. The ‘Ctrl+F’ Search (BM25 / Keywords)

This is the old-school search logic (like Elasticsearch or Lucene). It looks for the exact tokens you typed.

Superpower: Precision. If you search for ERR-502, it finds that exact string.

Kryptonite: Vocabulary mismatch. If you search ‘laptop screen,’ it might miss a document that only uses the word ‘display.’

The Fix: Hybrid Search

If you want a production-grade system, you cannot choose one. You need both. This is called Hybrid Search.

In a Hybrid architecture, every user query triggers two parallel searches:

Vector Search scans for semantic meaning.

BM25 Search scans for exact keywords.

You get two lists of results. One captures the intent, the other captures the specifics.

The ‘Merging’ Problem

Now you have a new engineering problem.

Vector search returns a Cosine Similarity score (usually 0.0 to 1.0).

BM25 returns a Relevance Score (which can be 5.0, 15.0, or 42.0 depending on document length and frequency).

You cannot compare these numbers. A 0.8 in vectors is not ‘better’ or ‘worse’ than a 12.0 in BM25. They are apples and oranges.

Enter Reciprocal Rank Fusion (RRF)

The solution is to ignore the scores and look at the rank.

Reciprocal Rank Fusion (RRF) doesn’t care that Document A got a score of 0.89. It cares that Document A was the #1 result. It takes the rankings from both lists and fuses them into a new, unified score.

If a document appears at the top of both lists, it skyrockets to the top of the final list. If it appears in only one, it gets pushed down.

But… What is a ‘Good’ RRF Score?

Congratulations, you have implemented Hybrid Search! But you have introduced a final, tricky variable.

RRF outputs abstract scores that look like 0.0163 or 0.032.

Unlike Cosine Similarity, where we know 0.85 is usually ‘good,’ RRF scores are unintuitive.

Is 0.016 a strong match or noise?

At what threshold should you cut off the results to prevent hallucinations?

Most developers guess. They pick a random number like 0.02 and hope for the best. Do not do that.

There is a mathematical way to calculate exactly where the ‘Noise Floor’ ends and ‘High Confidence’ begins.

👉 Read the sequel: Stop Guessing – A Mathematical Approach to RRF Thresholds in Hybrid Search

If you are building a RAG pipeline, you have probably hit this wall: Hybrid Search is great, but the scores are meaningless.

You combine your Dense Vector Search (Gemini/OpenAI) with your Sparse Keyword Search (BM25) using Reciprocal Rank Fusion (RRF). It works beautifully to bubble up the best results. But when you try to filter out the noise, you’re left staring at scores like 0.0163 or 0.0327.

Most tutorials tell you to “pick a threshold like 0.02.” But why? Why not 0.015? Why not 0.025?

We recently ran a deep dive into our own retrieval metrics to stop guessing and start calculating. Here is how we derived a mathematically justified threshold for RRF and solved the edge cases that break it.

The Problem: The “Magic Number”

RRF is a rank-based formula. It doesn’t care about cosine similarity or term frequency; it only cares about order.

(Standard k is usually 60)

Because the denominator is huge, the resulting scores are tiny.

• Rank 1 result: ~0.016
• Rank 10 result: ~0.014

When you merge two lists (Semantic + Keyword), the scores get summed. The problem is that a “mediocre” result from two sources looks mathematically similar to a “great” result from just one. We needed a way to distinguish Consensus from Noise.

The Research: Decoding the Score

We analyzed query logs across three distinct categories: Specific (perfect matches), Vague (conceptual matches), and Garbage (keyboard smashing).

We found a distinct “Ceiling” in the garbage results:

Query Type	Top RRF Score	Pattern
Specific	~0.032	High confidence from both algorithms.
Garbage	~0.016	Maxed out at exactly 0.0164.

The “Noise Floor” (0.016)

Why did garbage queries hit a wall at 0.016? The math explains it.

If Method A (e.g., Vector) thinks a document is Rank #1, but Method B (Keyword) doesn’t find it at all:

A score of ~0.016 represents Single-Source Confidence. It means one algorithm liked it, but the other one didn’t care. In a Hybrid system, this is often a hallucination or a partial match.

The “Consensus Floor” (0.025)

Now, look at what happens when both algorithms agree that a document is relevant (say, top 10 in both):

This gave us our mathematically derived threshold.

• Score >= 0.025: The document appeared in the Top 10 of BOTH methods. We have Consensus.
• Score < 0.016: Neither method ranked it highly. Noise.
• The Middle (0.016 – 0.025): The “Danger Zone” where only one algorithm is confident.

By setting our threshold to 0.025, we aren’t just picking a number. We are enforcing a policy: “For a result to be shown, both the semantic model and the keyword model must independently agree it is top-tier.”

The “Quantum Blockchain” Edge Case

There was one fatal flaw in our logic.

We ran the query: “quantum blockchain banana”.

• BM25: 0 results (Words didn’t exist in our docs).
• Vector: Found “nearest neighbors” with 0.53 similarity.
• RRF Score: ~0.016 (Single source).

Our new filter correctly blocked it! But wait! What if the user searches for a concept that uses no matching keywords?

If BM25 returns zero results, it’s a signal. It means there is zero lexical overlap. In this scenario, RRF breaks because it relies on two votes. When one voter is silent, the score naturally drops below our consensus threshold.

The Solution: The Dual-Filter Strategy

We realized we need different logic when lexical overlap is impossible.

1. If Keywords Match: Use RRF with the Consensus Threshold (0.025). We demand agreement.
2. If Keywords Fail: Fall back to Vector-only, but with a Strict Similarity Threshold (0.65).

If we can’t match your words, the meaning must be overwhelming for us to show a result.

The Code

Here is the logic we implemented to sanitize our RAG inputs:

Python

MIN_RRF_SCORE = 0.025  # Requires Top-10 ranking in BOTH methods
MIN_FAISS_SCORE = 0.65 # Requires strong semantic match if keywords fail

def smart_hybrid_search(query):
    bm25_results = get_sparse(query)
    faiss_results = get_dense(query)

    if bm25_results:
        # Standard Path: Demand Consensus
        merged = rrf_fusion(bm25_results, faiss_results)
        return [doc for doc in merged if doc.score >= MIN_RRF_SCORE]
    else:
        # Fallback Path: Strict Semantics
        # "Zero keyword overlap? The meaning better be exact."
        return [doc for doc in faiss_results if doc.score >= MIN_FAISS_SCORE]

Conclusion

Stop treating hybrid search thresholds as magic numbers.

• 0.016 is the sound of one hand clapping (Single Source).
• 0.025 is the sound of applause (Consensus).

By aligning our thresholds with the mathematical reality of RRF, we turned our retrieval layer from a “best guess” engine into a precision instrument.

---

Building Agents?