Skip to main content
Tutorial: Running inference with Kimi K2.7 Code using CoreWeave Inference (Serverless)
Getting set up and running Kimi K2, MoonShot AI's advanced long-context language model, in Python using CoreWeave Inference (Serverless). We'll be working with the moonshotai/Kimi-K2-Instruct model.
Last updated: October 24, 2026
Announcement: Kimi K2.7 Code is now live on CoreWeave Serverless Inference. Artificial Analysis places CoreWeave in the top speed-versus-price quadrant for the model, at roughly 289 tokens per second. Read the announcement.
💡
Kimi K2.7 Code is a coding-focused, agentic model from Moonshot AI, built for complex, long-horizon software engineering work. This tutorial runs it on CoreWeave Inference (Serverless) a serverless, OpenAI-compatible endpoint that hosts open-source models on CoreWeave infrastructure. You bring one Forge API key; you do not sign up for a separate hosting provider or stand up your own GPUs.
The point worth stating up front: inference, tracing, and evaluation live in the same place. The model runs on CoreWeave Inference (Serverless), and the same W&B project captures every call as a Weave trace and aggregates your evaluation runs. You send code context, prompts, tool-call instructions, images, or JSON schemas to the endpoint and get back structured, executable artifacts: functions, patches, test files, or schema-valid JSON.
This is a learn-by-doing sequence with checkpoints you can verify in your terminal and in Weave: environment verification, a first traced call, generated code that passes tests, schema-validated JSON, multi-turn context behavior, and a Weave evaluation you can share. Weave instrumentation starts at the first call, so every step is traceable from the beginning rather than added later.
By the end you will have a workflow you can hand to a teammate: a Weave evaluation page with per-example traces, inputs, generated code, pytest output, token usage, and configuration.
Table of contents
What changed from Kimi K2.6 for coding workloadsStep 0: Prerequisites and environmentStep 1: Validate the connection with a traced hello-code callStep 2: Generate code and execute the resultStep 3: Request structured JSON output, with optional vision inputStep 4: Preserve context across a multi-turn sessionStep 5: Evaluate Kimi K2.7 Code with Weave EvaluationsStep 6: Publish the evaluation as a shareable artifactPoint the same code at another endpointIf you self-host insteadExtend this to your own repo agentSources
What changed from Kimi K2.6 for coding workloads
Kimi K2.7 Code retains the K2 family shape (a 1 trillion-parameter total Mixture-of-Experts with roughly 32 billion active parameters per token) but is tuned more directly for code and agent workflows. Reported deltas versus K2.6 on coding and agentic benchmarks include about +21.8% on Kimi Code Bench v2, +11.0% on Program Bench, and +31.5% on MLS Bench Lite, with around 30% fewer reasoning tokens for comparable tasks.
Fewer reasoning tokens matter because cost and latency in agentic loops accumulate across long chains of tool calls and re-reads. Shorter thoughts mean fewer output tokens to generate, fewer tokens held in the growing context, and fewer cached token re-reads in later steps. The unit that matters in a CI bot or repo agent is the cost per merged pull request or per fixed test, not the cost per thousand tokens.
Capabilities relevant to code inference: a large context window in the 256k–262.1k token range for sustained multi-turn sessions, a configurable thinking mode, multi-turn tool calling, image input when the endpoint supports it, and structured outputs that adhere to a JSON schema.
Treat the benchmark numbers and reasoning-token figures as reported by the model provider, and verify them against your own workload using the evaluation steps near the end of this tutorial. Measured proof on your tasks beats a published delta.
Step 0: Prerequisites and environment
The CoreWeave Inference (Serverless) path needs no local GPU. You need Python, a W&B account with Inference credits, and a API key.
- CoreWeave Inference (Serverless) credits allocated to your account (included with Free, Pro, and Academic plans for a limited time; see the CoreWeave Inference (Serverless) pricing page)
- A Forge team name (team) and project, written as your-team/your-project
Python and pinned packages used in this tutorial:
- Python 3.11.x (Weave requires 3.9+)
- openai==1.30.5
- weave==0.50.0
- wandb==0.17.6
- pydantic==2.7.1
- pandas==2.2.2
- python-dotenv==1.0.1
- pytest==8.2.2
Install with one command:
python -m pip install --upgrade \openai==1.30.5 weave==0.50.0 wandb==0.17.6 \pydantic==2.7.1 pandas==2.2.2 python-dotenv==1.0.1 pytest==8.2.2
Set two environment variables. The same your-team/your-project string is used by both Weave (for tracing) and the Inference client (for usage tracking).
- WANDB_PROJECT: your-team/k27-code-inference
Run this environment check before any model call. It verifies imports, prints versions, and confirms the API path needs no GPU.
import os, sysfrom importlib.metadata import version, PackageNotFoundErrordef v(name):try:return version(name)except PackageNotFoundError:return "not-installed"for pkg in ["openai", "weave", "wandb", "pydantic", "pandas", "python-dotenv", "pytest"]:print(f"{pkg}-version", v(pkg))print("python-version", sys.version.split()[0])# Optional torch probe; the W&B Inference path does not need itgpu = Falsetry:import torchgpu = torch.cuda.is_available()print("torch-version", torch.__version__)except Exception:print("torch-version", "not-installed")print("api-path-requires-gpu=False")print("cuda-available", gpu)print("wandb-api-key-set", bool(os.getenv("WANDB_API_KEY")))print("wandb-project", os.getenv("WANDB_PROJECT") or "MISSING")
Expected output on a clean Python 3.11 install with the pins above:
openai-version 1.30.5weave-version 0.50.0wandb-version 0.17.6pydantic-version 2.7.1pandas-version 2.2.2python-dotenv-version 1.0.1pytest-version 8.2.2python-version 3.11.9torch-version not-installedapi-path-requires-gpu=Falsecuda-available Falsewandb-api-key-set Truewandb-project your-team/k27-code-inference
Step 1: Validate the connection with a traced hello-code call
The first call should already be observable. weave.init() autopatches the OpenAI client, so every chat.completions.create call is logged to your W&B project as a trace (inputs, outputs, latency, token usage, and cost) with no extra plumbing. The @weave.op decorator adds a span for your own function so you can see your code's inputs and outputs alongside the model call.
This is the canonical setup from the model page. Note the three things specific to CoreWeave Inference (Serverless): the base URL, the API key, and the project argument used for usage tracking.
import osimport weavefrom openai import OpenAIfrom dotenv import load_dotenvload_dotenv()PROJECT = os.environ["WANDB_PROJECT"] # "your-team/k27-code-inference"WANDB_API_KEY = os.environ["WANDB_API_KEY"] # from https://wandb.ai/authorizeMODEL_ID = os.getenv("KIMI_MODEL_ID", "moonshotai/Kimi-K2.7-Code")# Weave autopatches OpenAI and logs every call to this project.weave.init(PROJECT)client = OpenAI(base_url="https://api.inference.wandb.ai/v1", # W&B Inference, powered by CoreWeaveapi_key=WANDB_API_KEY, # your W&B API keyproject=PROJECT, # team/project for usage tracking)@weave.op()def k27_chat(messages, temperature=0.2, top_p=0.95, max_tokens=256):resp = client.chat.completions.create(model=MODEL_ID,messages=messages,temperature=temperature,top_p=top_p,max_tokens=max_tokens,)return resp.choices[0].message.contentanswer = k27_chat([{"role": "system", "content": "You are a precise Python coding assistant."},{"role": "user", "content": "Write a Python one-liner that returns the squares of 1..5 as a list."},])print("model", MODEL_ID)print("answer", answer.strip())print("View the trace under the Traces tab at https://wandb.ai/" + PROJECT)
Example observed output:
model moonshotai/Kimi-K2.7-Codeanswer squares = [i*i for i in range(1, 6)]View the trace under the Traces tab at https://wandb.ai/your-team/k27-code-inference
Open the Traces tab in your W&B project and you will see this call: the prompt, the response, latency, and the token usage that Weave captured automatically. Your wording may differ from the example.
Confirm the exact model slug on the model page (https://wandb.ai/inference). The API expects the Hugging-Face-style org/model slug shown there, for example moonshotai/Kimi-K2.7-Code.
Step 2: Generate code and execute the result
A useful code-inference test produces code and then runs it. We ask Kimi K2.7 Code to implement an interval-merging function, return only a fenced Python block, extract it, write deterministic pytest tests, and run them.
import re, subprocess, tempfile, textwrapfrom pathlib import Pathimport weave@weave.op()def generate_merge_intervals():prompt = ("Implement merge_intervals(intervals: list[tuple[int,int]]) -> list[tuple[int,int]].\n""Requirements:\n""- Merge overlapping or touching intervals.\n""- Return intervals sorted by start.\n""- Output only a single fenced Python code block with the function and any imports.")resp = client.chat.completions.create(model=MODEL_ID,messages=[{"role": "system", "content": "You are a precise Python coding assistant."},{"role": "user", "content": prompt},],temperature=0.2, top_p=0.95, max_tokens=400,)txt = resp.choices[0].message.contentm = re.search(r"```python(.*?)```", txt, re.S | re.I) or re.search(r"```(.*?)```", txt, re.S)code = m.group(1).strip() if m else txt.strip()return {"raw": txt, "code": code}out = generate_merge_intervals()print("generated-snippet-head", out["code"].splitlines()[0][:80])with tempfile.TemporaryDirectory() as td:td = Path(td)(td / "solution.py").write_text(out["code"])tests = textwrap.dedent("""from solution import merge_intervalsdef test_basic():assert merge_intervals([(1,3),(2,4),(6,7)]) == [(1,4),(6,7)]def test_touching():assert merge_intervals([(1,2),(3,5),(2,3)]) == [(1,5)]def test_nested():assert merge_intervals([(1,10),(2,3),(4,8)]) == [(1,10)]def test_single():assert merge_intervals([(5,5)]) == [(5,5)]def test_empty():assert merge_intervals([]) == []""").strip()(td / "test_solution.py").write_text(tests)proc = subprocess.run(["pytest", "-q"], cwd=td, capture_output=True, text=True)print(proc.stdout.strip())print("pytest-exit-code", proc.returncode)
Example output:
generated-snippet-head def merge_intervals(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]:..... [100%]5 passed in 0.06spytest-exit-code 0
This is the first task-level checkpoint. Success means the output compiled and passed tests, not that it read plausibly. The call is in your Traces tab alongside its token usage.
Step 3: Request structured JSON output, with optional vision input
Structured outputs turn model text into an agent contract you can machine-check. We request a change plan that validates against a Pydantic schema, and show an optional image variant for endpoints that accept image inputs through the OpenAI-compatible API.
import osfrom typing import Literal, Listfrom pydantic import BaseModel, ValidationErrorimport weaveVISION_ENABLED = os.getenv("VISION_ENABLED", "false").lower() == "true"VISION_IMAGE_URL = os.getenv("VISION_IMAGE_URL", "")class ChangePlan(BaseModel):files_changed: List[str]tests_to_run: List[str]risk_level: Literal["low", "medium", "high"]rationale: str@weave.op()def structured_change_plan():base_text = ("You are a repo planning assistant. Propose a small change plan for adding a --json flag to a CLI tool.\n""Return a JSON object with fields: files_changed: str[], tests_to_run: str[], ""risk_level: 'low'|'medium'|'high', rationale: string.\n""Do not include any text outside the JSON.")if VISION_ENABLED and VISION_IMAGE_URL:user_content = [{"type": "text", "text": base_text + "\nConsider the attached CLI screenshot."},{"type": "image_url", "image_url": {"url": VISION_IMAGE_URL}},]else:user_content = base_textresp = client.chat.completions.create(model=MODEL_ID,messages=[{"role": "system", "content": "You are a precise repo agent."},{"role": "user", "content": user_content},],response_format={"type": "json_object"},temperature=0.2, top_p=0.95, max_tokens=300,)raw = resp.choices[0].message.contenttry:parsed = ChangePlan.model_validate_json(raw)return {"schema_valid": True, "parsed": parsed.model_dump(),"vision_used": bool(VISION_ENABLED and VISION_IMAGE_URL)}except ValidationError as e:return {"schema_valid": False, "error": str(e), "raw": raw}import jsonout = structured_change_plan()print("schema-valid", out["schema_valid"])if out["schema_valid"]:print("vision-used", out["vision_used"])print(json.dumps(out["parsed"], indent=2))else:print("validation-error", out["raw"][:240])
Example output without an image configured:
schema-valid Truevision-used False{"files_changed": ["cli/main.py", "cli/args.py", "tests/test_cli_json.py"],"tests_to_run": ["pytest -q tests/test_cli_json.py"],"risk_level": "low","rationale": "Add a --json flag parsed via argparse, return machine-readable output, keep default text behavior unchanged."}
Schema-valid output like this is what lets you wire tool calling, CI bots, and eval scorers, because you check the result instead of scraping prose.
Step 4: Preserve context across a multi-turn session
Long-horizon coding agents depend on context persistence. We carry a sentinel detail across two turns to confirm the second response used the first. Thinking behavior is provider-specific: some endpoints expose a deeper-thinking toggle. Do not assume identical parameter names across providers.
import timeimport weave@weave.op()def multiturn_session():sentinel = "ACME-TRACE-42"sysmsg = "You are a careful Python library maintainer."turn1 = [{"role": "system", "content": sysmsg},{"role": "user", "content": f"Our internal HTTP client adds header X-Trace-Client: {sentinel}. ""Expose a client.get(url) helper that preserves headers and retries on 429."},]r1 = client.chat.completions.create(model=MODEL_ID, messages=turn1,temperature=0.2, top_p=0.95, max_tokens=300)m1 = r1.choices[0].message.contentturn2 = turn1 + [{"role": "assistant", "content": m1},{"role": "user", "content": "Write a concise docstring for client.get explaining header behavior. ""Only output the docstring text."},]r2 = client.chat.completions.create(model=MODEL_ID, messages=turn2,temperature=0.2, top_p=0.95, max_tokens=120)m2 = r2.choices[0].message.contentused_prior = sentinel in m2 or "X-Trace-Client" in m2return {"used_prior_context": used_prior, "turn2_text": m2.strip()}out = multiturn_session()print("used-prior-context", out["used_prior_context"])print("turn2-text-head", out["turn2_text"][:120].replace("\n", " "))
Example output:
used-prior-context Trueturn2-text-head client.get(url) -> Response Performs an HTTP GET with exponential backoff on 429 and always sends the X-Trace-Client: ACME-TRACE-42 header.
The model kept the sentinel header across turns. For long sessions, budget context: the 256k–262.1k window allows long histories, but cached input still costs when the model rereads long transcripts in agent loops. Both turns appear in your Traces tab with their token counts.
Step 5: Evaluate Kimi K2.7 Code with Weave Evaluations
A small evaluation turns one-off calls into a repeatable measurement. We build a four-task dataset mixing code execution, JSON validation, and repository-context planning, then run it through a weave.Evaluation.
The Weave evaluation pattern has three parts: a weave.Model subclass whose @weave.op predict method produces output, one or more @weave.op scorer functions that receive the dataset fields plus the model output, and a weave.Evaluation(dataset=, scorers=) you run with asyncio.run(evaluation.evaluate(model)). Token usage, latency, and cost are captured automatically by Weave on each traced call and aggregated in the comparison view, so the scorers only need to return pass/fail signal.
import re, json, asyncio, subprocess, tempfile, textwrapfrom pathlib import Pathfrom typing import Optionalimport weavefrom pydantic import BaseModel# Reuse the same client/init from the hello-code step (weave.init + W&B Inference client).dataset = [{"id": "bugfix-offbyone", "kind": "exec", "prompt": ("Fix this function so it returns the first index of target, or -1.\n""Output only a fenced Python code block defining find_index.\n\n""def find_index(arr, target):\n"" for i in range(len(arr)):\n"" if arr[i] == target:\n"" return i\n"" return 1 # BUG: should be -1\n")},{"id": "tests-for-sum-json", "kind": "json", "prompt": ("Given a module mathx.py with function add(a:int,b:int)->int, propose tests.\n""Return JSON with fields: tests: [{name, code}]. No text outside JSON.")},{"id": "refactor-to-schema", "kind": "json", "prompt": ("We will add a config file for a CLI. Return JSON with fields: ""version: string, options: [{name, type, default}], rationale: string. Only JSON.")},{"id": "repo-change-plan", "kind": "repo", "prompt": ("You maintain a small library: core/__init__.py, core/http.py, cli/main.py, tests/test_http.py.\n""Propose a minimal change to add request timeout handling.\n""Return JSON {files_changed, tests_to_run, risk_level, rationale}. Only JSON.")},]class PlanSchema(BaseModel):files_changed: list[str]tests_to_run: list[str]risk_level: strrationale: strclass K27CodeModel(weave.Model):model_id: str@weave.op()def predict(self, prompt: str, kind: str) -> dict:resp = client.chat.completions.create(model=self.model_id,messages=[{"role": "system", "content": "You are a precise coding assistant."},{"role": "user", "content": prompt},],temperature=0.2, top_p=0.95, max_tokens=500,response_format={"type": "json_object"} if kind in ("json", "repo") else None,)return {"text": resp.choices[0].message.content}@weave.op()def code_executes(kind: str, output: dict) -> dict:if kind != "exec":return {"applicable": False, "pass": None}txt = output["text"]m = re.search(r"```python(.*?)```", txt, re.S | re.I) or re.search(r"```(.*?)```", txt, re.S)code = (m.group(1).strip() if m else txt).strip()with tempfile.TemporaryDirectory() as td:tdp = Path(td)(tdp / "solution.py").write_text(code)(tdp / "test_solution.py").write_text(textwrap.dedent("""from solution import find_indexdef test_hit(): assert find_index([1,2,3],2) == 1def test_miss(): assert find_index([1,2,3],4) == -1""").strip())proc = subprocess.run(["pytest", "-q"], cwd=tdp, capture_output=True, text=True)return {"applicable": True, "pass": proc.returncode == 0,"detail": (proc.stdout.strip().splitlines() or ["no output"])[-1]}@weave.op()def valid_json(kind: str, output: dict) -> dict:if kind not in ("json", "repo"):return {"applicable": False, "pass": None}try:if kind == "repo":PlanSchema.model_validate_json(output["text"])else:json.loads(output["text"])return {"applicable": True, "pass": True}except Exception as e:return {"applicable": True, "pass": False, "detail": str(e)[:200]}model = K27CodeModel(model_id=MODEL_ID)evaluation = weave.Evaluation(name="k27-mini-eval", dataset=dataset,scorers=[code_executes, valid_json])# Cap parallelism so async evaluation does not trip endpoint rate limits.import osos.environ.setdefault("WEAVE_PARALLELISM", "3")summary = asyncio.run(evaluation.evaluate(model))print(json.dumps(summary, indent=2, default=str))print("Open the Evals tab at https://wandb.ai/" + PROJECT + " to compare runs.")
Example summary (shape will vary):
{"code_executes": {"pass": {"true_count": 1, "true_fraction": 1.0}},"valid_json": {"pass": {"true_count": 3, "true_fraction": 1.0}},"model_latency": {"mean": 1.02},"usage": {"total_tokens": {"mean": 187.5}}}Open the Evals tab at https://wandb.ai/your-team/k27-code-inference to compare runs.
This answers the practical question: does K2.7 Code complete small coding tasks within your latency and cost envelope. Each example is a row in the Weave Evals view with its prompt, generated code, scorer result with pytest output, latency, and token usage. You now have a baseline pass rate rather than single-prompt impressions.
For a rough cost figure, apply reported K2.7 Code pricing (0.95 USD per 1M input tokens, 4.00 USD per 1M output tokens, and 0.19 USD per 1M cache hits where applicable) to the token usage Weave records. Confirm current rates on the CoreWeave Inference (Serverless) pricing page.
Step 6: Publish the evaluation as a shareable artifact
The artifact your teammate opens is the Weave Evals comparison page, no rerun required. Because you ran the evaluation in your W&B project, it is already there. Open the Evals tab at https://wandb.ai/your-team/k27-code-inference, select the k27-mini-eval run, and use the share control in the W&B UI to send the link.
Configuration is captured for you: the model_id is recorded as a field on the weave.Model, each scorer call is a trace, and the OpenAI calls carry their own token usage and cost. To track multiple configurations, give each weave.Evaluation a distinct name (for example k27-mini-eval-thinking-on) or vary the K27CodeModel fields, then compare them side by side in the Evals view.
This page becomes your baseline for comparing K2.7 Code configurations, a K2.6 migration, or different prompt and thinking settings.
Point the same code at another endpoint
CoreWeave Inference (Serverless) uses the standard OpenAI Chat Completions interface, so the wrapper above is portable. To compare against another OpenAI-compatible endpoint, change three things (base_url, api_key, and the model slug) and rerun the same evaluation. Your messages, response_format, tool schemas, and scorers stay the same.
# Same wrapper, different endpoint. W&B Inference is the default here.client = OpenAI(base_url=os.environ["KIMI_BASE_URL"],api_key=os.environ["KIMI_API_KEY"],)resp = client.chat.completions.create(model=os.environ["KIMI_MODEL_ID"],messages=[{"role": "user", "content": "Return the string 'ok'."}],)print(resp.choices[0].message.content.strip())
Because Weave traces whichever endpoint the OpenAI client points at, you can run the mini evaluation against each and compare cost per completed task and latency in the same project.
If you self-host instead
If you need to run the weights yourself, serving Kimi K2.7 Code is a memory-planning exercise. You take on the GPUs, memory, and scaling that CoreWeave Inference (Serverless) otherwise handles. Reported storage and memory figures for planning:
- Full-precision weights are about 605 GB on disk.
- Q8 packages are around 595 GB, nearly indistinguishable in perplexity from full precision for most uses.
- Dynamic 2-bit (UD-Q2) variants land in the low-to-mid 300 GB range (commonly ~325–345 GB), with a favorable quality-to-size tradeoff for code.
- Q4 sits between Q2 and Q8 with small reported quality loss and a smaller footprint than Q8.
- Dynamic 1-bit is experimental and provider-dependent; validate it against your own evaluation before production use.
Match combined RAM plus VRAM to the quant size plus headroom for KV cache and runtime buffers. Multi-GPU serving typically relies on tensor parallelism and high-bandwidth interconnects, and effective throughput depends on target context length and batching.
import mathdef plan_self_host(quant_size_gb: float, gpu_mem_gb: float, num_gpus: int | None,kv_cache_gb: float, reserve_system_gb: float = 32.0):"""Feasibility check: require sum(VRAM) >= quant_size_gb + kv_cache_gb."""min_gpus = math.ceil((quant_size_gb + kv_cache_gb) / gpu_mem_gb)gpus_ok = (num_gpus or min_gpus) * gpu_mem_gb >= quant_size_gb + kv_cache_gbreturn {"min_gpus_required": min_gpus,"gpus_ok_with_num_gpus": (num_gpus is not None and gpus_ok),"total_vram_needed_gb": round(quant_size_gb + kv_cache_gb, 1),"notes": "Add headroom for runtime buffers and concurrency.",}print("UD-Q2 plan:", plan_self_host(345, 80, None, 40))print("Q4 plan:", plan_self_host(420, 80, None, 40))print("Q8 plan:", plan_self_host(595, 80, None, 60))
Example output:
UD-Q2 plan: {'min_gpus_required': 5, 'gpus_ok_with_num_gpus': False, 'total_vram_needed_gb': 385.0, 'notes': 'Add headroom for runtime buffers and concurrency.'}Q4 plan: {'min_gpus_required': 6, 'gpus_ok_with_num_gpus': False, 'total_vram_needed_gb': 460.0, 'notes': 'Add headroom for runtime buffers and concurrency.'}Q8 plan: {'min_gpus_required': 9, 'gpus_ok_with_num_gpus': False, 'total_vram_needed_gb': 655.0, 'notes': 'Add headroom for runtime buffers and concurrency.'}
For vLLM, set tensor-parallel-size to your GPU count, max-model-len to your target context (up to 262,144 for full long context), dtype to match your quantization package, and concurrency caps to keep KV cache within VRAM. Rerun the mini evaluation to confirm code-task quality before adopting any aggressive quantization.
Extend this to your own repo agent
Acceptance criteria: the Weave evaluation is visible in your project, every Kimi K2.7 Code call is traced, generated code passes tests, structured outputs validate, and your configuration is logged. Treat the four-task evaluation as a smoke test. The measurement that matters comes from your own repository tasks, CI failures, or migration backlog, where cost per completed task and reliability decide the outcome.
One realistic extension: connect the traced wrapper to a repo agent that opens pull requests and appends each new failing case to your Weave evaluation set, so your dashboard grows with production-shaped failures, all running on CoreWeave Inference (Serverless), in one project.
Sources
Add a comment