Pinecone

You decide to move from `text-embedding-ada-002` to `text-embedding-3-large`, or from OpenAI to a Cohere model, and suddenly your Pinecone index is a liability. The new model outputs 3072 dimensions, your index was created at 1536, and vectors from two different models are not comparable, so you cannot mix them in one namespace. The naive path is to delete, recreate, and re-upsert everything, which means your production RAG endpoint returns garbage for the entire backfill window.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset, with Knowledge Bases as the second surface for unstructured sources like PDFs and support docs. It matters here because a model migration is only painless if you have a clean, queryable source of truth to re-embed FROM, one that emits a change event the moment a document is edited.

This article walks the Pinecone-native mechanics first: dimension mismatches, the blue-green namespace swap, and batched backfill without dropping reads. Then it shows how sourcing your embedding input from Sanity Context turns a scary one-shot migration into a boring, incremental one.

Why can't I just re-embed into my existing Pinecone index?

You cannot re-embed into an existing Pinecone index because the index dimension is fixed at creation time, and different embedding models produce vectors of different dimensions that live in incompatible spaces. An index created for `text-embedding-ada-002` is 1536 dimensions. `text-embedding-3-large` is 3072. If you try to upsert a 3072-dim vector into a 1536-dim index, Pinecone rejects it outright:

`PineconeException: Vector dimension 3072 does not match the dimension of the index 1536`

Even when two models happen to share a dimension count, the numbers mean nothing across models. A cosine similarity of 0.82 between an ada vector and a `3-large` vector is noise. The geometry is different, so a query embedded with the new model and compared against old vectors returns nonsense rankings. This is the trap teams fall into when they swap the embedding call in their ingestion code but keep upserting into the same namespace: for a while, the index is a mix of two models, and every query is quietly wrong.

The correct mental model is that a Pinecone index (or at minimum a namespace) is bound to exactly one embedding model. Changing models is not an in-place edit. It is a rebuild. The whole game is rebuilding the new copy alongside the live one so reads never hit a half-populated index. That is what the rest of this article sets up: a blue-green swap where the old index keeps serving traffic until the new one is fully backfilled and verified.

from pinecone import Pinecone

pc = Pinecone(api_key="...")
index = pc.Index("docs-ada-1536")

# new model returns 3072 dims
vec = embed_with_3_large("some document text")  # len(vec) == 3072

index.upsert(vectors=[("doc-1", vec, {"source": "docs"})])
# PineconeException: Vector dimension 3072 does not match
# the dimension of the index 1536

How do I do a blue-green namespace swap in Pinecone?

A blue-green swap in Pinecone means you build the new-model index in parallel, keep serving reads from the old one, and cut over only when the new one is complete and verified. With serverless indexes this is cheap because you pay for what you store and query, so running two indexes for a few hours during migration costs very little.

The cleanest version uses two separate indexes, one per model, because the dimensions differ and a single index cannot hold both. Create `docs-3large-3072` alongside the live `docs-ada-1536`. Your application reads from whichever index name an environment variable or config flag points at. The backfill job writes only to the new index. Queries never touch a partially built index because your read path still resolves to the old one until you flip the flag.

If you are staying on the same model dimension and only re-chunking or refreshing content, you can instead use two namespaces inside one index (`v1` and `v2`) and swap the namespace your query path targets. Namespaces are isolated, so a query against `v2` never sees `v1` vectors. Either way, the invariant is the same: reads resolve to a fully populated target, writes go to the target being built, and the flip is a single config change you can roll back instantly by pointing the flag back at the old target.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="...")

# build the new-model index alongside the live one
if not pc.has_index("docs-3large-3072"):
    pc.create_index(
        name="docs-3large-3072",
        dimension=3072,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1"),
    )

# read path resolves the active index from config, not hardcoded
ACTIVE_INDEX = os.environ["ACTIVE_INDEX"]  # "docs-ada-1536" until cutover
read_index = pc.Index(ACTIVE_INDEX)

# backfill path always writes to the new index
write_index = pc.Index("docs-3large-3072")

How do I backfill millions of vectors without rate-limiting myself?

You backfill without rate-limiting by batching upserts, embedding concurrently but capping in-flight requests, and paginating your source instead of loading it all into memory. Pinecone accepts up to 1000 vectors per upsert call for smaller vectors, and fewer for high-dimension ones, so batch in the low hundreds when you are on 3072 dimensions to stay under the request size limit.

The bottleneck is almost never Pinecone. It is your embedding provider's rate limit. OpenAI's embedding endpoints are generous but not infinite, and a naive `for` loop that awaits each embed serially will take hours you do not need to spend. Use a bounded concurrency pool, something like 10 to 20 concurrent embed calls, then group the results into upsert batches. If you hit a 429 from the embedding provider, back off exponentially and retry the batch rather than the whole job.

Target namespaces deliberately during backfill. Write everything to the new index, and if you are versioning within one index, write to the `v2` namespace so a stray read against `v1` never returns a half-migrated record. Track progress with a durable cursor (the last document ID you completed) so a crashed job resumes instead of restarting. The point of all this plumbing is that the live index keeps answering queries at full quality the entire time. Nobody querying your RAG endpoint during the backfill sees a single degraded result, because their reads are pinned to the old index until you flip the flag.

import asyncio
from pinecone import Pinecone

pc = Pinecone(api_key="...")
write_index = pc.Index("docs-3large-3072")
sem = asyncio.Semaphore(15)  # cap in-flight embed calls

async def embed_doc(doc):
    async with sem:
        vec = await embed_3_large(doc["text"])  # 3072 dims
        return (doc["id"], vec, {"source": doc["source"]})

async def backfill(docs, batch=200):
    for i in range(0, len(docs), batch):
        chunk = docs[i : i + batch]
        vectors = await asyncio.gather(*[embed_doc(d) for d in chunk])
        write_index.upsert(vectors=vectors, namespace="v2")
        save_cursor(chunk[-1]["id"])  # resume point on crash

What should I re-embed FROM so the backfill stays correct?

You should re-embed from a single canonical source of truth that you can query deterministically and re-fetch on demand, not from whatever text you happened to stuff into Pinecone metadata last year. This is the step most migration guides skip, and it is where re-embedding jobs silently corrupt data. If the only copy of your document text lives in Pinecone metadata, your new index inherits every truncation, every stale edit, and every encoding bug from the old one. You are re-embedding drift.

Sanity Context solves the source-of-truth problem because the content already lives in a structured, versioned dataset you can query with GROQ, and you re-embed from that query rather than from your vector store. The primary surface, Context MCP, is a hosted read-only MCP endpoint, so an agent or a migration job can pull the exact current text of every document through schema-aware queries and reference traversal. Structured content (articles, product entries, anything with a schema) comes back through GROQ. Unstructured sources (PDFs, support databases, scraped websites) go through Knowledge Bases, which turns messy corpora into ordered documents with a clear table of contents, giving your backfill clean input instead of raw file dumps.

This is where Sanity earns the label Content Operating System for the AI era: it operates content end to end, so the same governed dataset that editors publish from is the dataset your re-embedding job reads, with no separate sync pipeline to drift out of alignment. Your Pinecone backfill loop stops pulling text out of vector metadata and starts pulling it from a GROQ query, which means the new index reflects what the content actually says today, not a snapshot frozen at first ingest.

import { createClient } from "next-sanity";

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: "production",
  apiVersion: "2024-01-01",
  useCdn: false,
});

// re-embed FROM the canonical, current text, paginated by _id
const docs = await sanity.fetch(
  `*[_type == "article" && _id > $cursor] | order(_id) [0...200]{
    "id": _id,
    "text": pt::text(body),
    "source": _type
  }`,
  { cursor }
);
// hand docs to embed_3_large(...) then upsert into docs-3large-3072

How do I keep the new index current DURING the migration?

You keep the new index current during migration by treating every content edit as a change event and re-embedding just that document, so the backfill and live editing do not race each other. The classic failure is a long backfill that takes six hours, during which editors publish 40 changes. Your new index now has 40 stale vectors the moment it goes live, because the backfill read those documents before they changed.

In Sanity, a document webhook fires on publish, which gives you a per-document change event to re-embed against. Wire that webhook to a small function that fetches the single changed document by ID, embeds it with the new model, and upserts it into the new index. Now edits during the backfill window are captured incrementally instead of lost. The migration stops being a one-shot snapshot and becomes a steady state: the new index converges on correct, and stays correct, without a second full pass.

This also solves the ongoing freshness problem that outlives the migration. Once you cut over, the same webhook keeps your Pinecone index in sync with published content, so an edit shows up in retrieval within seconds rather than waiting for a nightly rebuild. The Live Content API and Functions give you the event stream and the compute to run the re-embed close to the source. The migration plumbing you built is not throwaway. It is the permanent ingestion path.

// POST target for a Sanity publish webhook filtered to _type == 'article'
import { Pinecone } from "@pinecone-database/pinecone";

const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pc.index("docs-3large-3072");

export async function POST(req: Request) {
  const { _id, text, _type } = await req.json();
  const values = await embed3Large(text); // 3072 dims

  await index.namespace("v2").upsert([
    { id: _id, values, metadata: { source: _type } },
  ]);

  return new Response("ok");
}

When should I skip Pinecone and query Sanity Context directly?

You should skip the separate Pinecone index entirely when your retrieval query has a structural component that pure vector similarity cannot resolve, because Sanity Context can run semantic ranking and structural filtering in one GROQ query. If a user asks for "the latest pricing article for the enterprise plan," a pure vector search over Pinecone can match on "pricing" and "enterprise" but has no reliable way to enforce "latest" (a date order) or "published" (a state filter). You end up post-filtering vector results in application code and hoping enough survived the filter to fill the result set.

With Sanity Context, structural predicates are real query filters, and semantic ranking is a scoring step in the same query, so date ranges, author, product variant, and publication state are enforced before ranking, not after. Embeddings in Context are opt-in and off by default, and most projects shipping on Context MCP never turn them on because structured GROQ queries and schema lookups already answer the heavy majority of retrieval calls. Semantic search is the deeper layer you reach for when keyword and structural matching genuinely miss.

Pinecone still earns its place for high-volume, machine-generated corpora that do not need editorial governance, millions of log lines or product-telemetry embeddings that no human will ever edit or approve. The routing rule is simple: structured, governed content that editors own should be queried through Sanity Context; sprawling machine-generated vectors that need a dedicated ANN engine at scale belong in Pinecone. Many production stacks run both, and a model migration is a good moment to ask which of your indexes actually needed to be a separate vector database in the first place.