Knowledge Concepts
Embeddings are numerical representations of text, images, or other content — lists of numbers that encode meaning so that similar concepts produce similar numbers, enabling machine learning models to understand and compare content semantically.
Embeddings are numerical representations of content — text, images, audio, or code converted into lists of numbers (vectors) that encode meaning. The key property: semantically similar content produces numerically similar embeddings. "Heap overflow" and "memory exhaustion" produce embeddings that are close to each other in numerical space; "heap overflow" and "chocolate cake" produce embeddings far apart. This numerical encoding of meaning is the foundation of modern AI search, recommendation systems, and large language models.
When an AI "understands" that your question is related to a topic, it's measuring similarity between embeddings.
Early natural language processing (NLP) represented words as discrete symbols — each word was a unique identifier with no inherent relationship to other words. "Cat" and "feline" were as different as "cat" and "carburetor." This made semantic understanding impossible for machines.
Key milestones:
One-hot encoding (1980s-2000s): Each word represented as a binary vector: 0 everywhere except a single 1 at the word's index position. "Cat" might be [0,0,0,1,0,...,0] and "feline" [0,0,1,0,0,...,0]. No meaningful similarity — any two words were equally "distant."
Word2Vec (Mikolov et al., Google, 2013): The breakthrough: train a neural network to predict words from surrounding context, and the internal representations (embeddings) that emerge encode semantic relationships. "Cat" and "feline" end up with similar embeddings. Analogical reasoning like "king - man + woman ≈ queen" works because these relationships are geometrically encoded.
BERT and transformer-based contextual embeddings (2018): Word2Vec produced the same embedding for a word regardless of context. BERT (Bidirectional Encoder Representations from Transformers) produces different embeddings for the same word in different contexts — "bank" in "river bank" vs. "financial bank" gets different embeddings. This contextual understanding dramatically improved downstream tasks.
Modern sentence and paragraph embeddings: Models like OpenAI's text-embedding-ada-002, Cohere's Embed, and open-source models (Sentence-BERT, E5, BGE) produce embeddings for entire sentences or paragraphs rather than individual words — these are the embeddings used in modern semantic search and RAG systems.
The basic mechanism: An embedding model (a trained neural network) takes input content and produces a vector — a list of numbers, typically 768, 1024, or 1536 dimensions depending on the model. Each dimension encodes some aspect of the content's meaning, though the dimensions are not individually interpretable.
Example (simplified): "The server is running out of memory" → [0.023, -0.147, 0.891, 0.443, ..., -0.021] (1,536 numbers)
Similarity measurement: The semantic similarity between two embeddings is measured by cosine similarity — the angle between the two vectors in high-dimensional space. Vectors pointing in nearly the same direction have cosine similarity close to 1 (very similar). Perpendicular vectors have cosine similarity 0 (unrelated). Opposite vectors have cosine similarity -1 (antonyms).
What the numbers encode: The individual numbers in an embedding don't correspond to human-readable concepts ("dimension 347 = 'memory'"). Instead, the collective pattern of all dimensions encodes the content's position in a learned semantic space. Similar content ends up in similar regions of this space — which is what makes nearest-neighbor search for meaning possible.
A developer saves debugging notes over two years. They embed each note when they save it.
Notes and their approximate semantic neighborhoods:
When the developer searches "memory problem in backend services": The query is embedded → nearest-neighbor search finds embeddings close to the query vector.
Results ranked by similarity:
The search found "memory" notes even though neither used the phrase "memory problem" — because the semantic meaning was encoded in the embedding.
Word embeddings: Represent individual words. Word2Vec, GloVe, FastText. Simple, fast, useful as building blocks but limited — no context, no sentence-level understanding.
Sentence embeddings: Represent entire sentences or paragraphs as a single vector. Sentence-BERT, E5, text-embedding-ada-002. Most useful for semantic search and RAG applications.
Multimodal embeddings: Represent different content types in the same vector space — text and images together. CLIP (OpenAI) embeds images and text so "a photo of a dog" and [an actual dog photo] are close in embedding space. Enables cross-modal search.
Code embeddings:
Represent code in vector space. CodeBERT, UniXcoder. Find semantically similar code even when syntax differs — "sort array ascending" finds array.sort() in Python and Arrays.sort(arr) in Java.
Semantic search: Convert your document collection to embeddings (done once, when documents are added). Convert each search query to an embedding (done at search time). Find the stored embeddings most similar to the query embedding. Return those documents.
This is how modern knowledge management tools implement "AI search" — it's vector search over embeddings.
RAG (Retrieval-Augmented Generation): Convert your knowledge base to embeddings → store in a vector database → when an AI receives a question, embed the question → retrieve most similar knowledge base chunks → pass retrieved context to LLM as additional input → LLM generates grounded, specific response.
The embedding enables the retrieval step: finding which parts of your knowledge base are most relevant to the question.
Recommendation: Products you've interacted with → embed → find similar products → recommend. Users with similar behavior patterns → embed → find similar users → collaborate-filter recommendations. Embeddings are the foundation of most modern recommendation systems.
Classification and clustering: Cluster documents by embedding similarity to discover topic groups without predefined categories. Classify new documents by finding the nearest training examples in embedding space.
"Embeddings understand language the way humans do." Embeddings encode statistical patterns in training data — correlations between words and contexts. They're powerful for similarity detection and semantic tasks but don't "understand" in the way humans do. They fail on simple counting, explicit negation ("not blue"), and logical reasoning tasks that require symbol manipulation rather than pattern matching.
"All embeddings are the same." Embedding quality varies significantly by model and training data. An embedding model trained on code is better for code search; one trained on medical literature is better for medical semantic search. Multilingual models produce cross-lingual embeddings (same concept in different languages → similar vectors); monolingual models don't.
"You need to understand the math to use embeddings." Using embeddings as a developer means calling an API (OpenAI's embeddings API, Cohere's embed endpoint) that returns a vector, then storing and querying those vectors. The mathematical details (transformers, attention mechanisms, backpropagation) are abstracted away.
Vector database: The storage and search infrastructure for embeddings — stores vectors and enables fast nearest-neighbor search.
Full-text search: The complementary approach — finds exact keyword matches where embeddings find semantic similarity.
RAG (Retrieval-Augmented Generation): The AI architecture pattern that uses embeddings to retrieve relevant context before LLM generation.
Large language models: The model class that produces contextual embeddings as internal representations — and that RAG feeds retrieved context into.
How do I get embeddings for my documents?
Call an embedding API: OpenAI's text-embedding-ada-002 or text-embedding-3-small, Cohere's embed-english, or use an open-source model locally (Sentence-BERT via the sentence-transformers Python library). The API returns a vector for each text input.
How many dimensions should an embedding have? More dimensions generally (but not always) produce better quality at higher storage and computation cost. OpenAI's text-embedding-3-small produces 1,536 dimensions; text-embedding-3-large produces 3,072. For most applications, 1,024-1,536 dimensions is sufficient. Open-source models range from 384 to 4,096 dimensions.
Can embeddings encode images and text together? Yes — multimodal models like CLIP embed images and text in the same vector space. This enables text-to-image search: a query "sunset over mountains" finds photos with semantically similar content, even without any text metadata on the photos.
Embeddings are the bridge between human language and machine understanding — the numerical language that lets computers work with meaning rather than just symbols. For developers and knowledge workers, understanding embeddings explains how AI search finds relevant documents without exact keywords, how RAG grounds AI responses in specific knowledge, and why "AI that understands your notes" is technically achievable at relatively low cost. The implementation is straightforward; the conceptual shift — from searching for words to searching for meaning — is what opens up new possibilities.
More WebSnips articles that pair well with this topic.
Intermediate packets are small, discrete, reusable units of work — an outline, a research summary, a slide deck draft, a set of notes — that can be saved and reused across future projects rather than starting from scratch each time.
A bullet journal (BuJo) is an analog organization system created by Ryder Carroll using a dot-grid notebook: rapid logging with bullet symbols, collections (logs, trackers, lists), and migration (intentional task review). It combines daily planner, to-do list, diary, and habit tracker in one notebook.
A KWL chart is a three-column learning organizer: K (What I Know), W (What I Want to Know), L (What I Learned). It activates prior knowledge before learning, sets specific goals, and documents what you actually learned — making the learning process active and self-directed.