# Embeddings > An embedding is a list of numbers that represents the meaning of a text or image so similar things sit close together. How to compare them and what to avoid. *Tags: ai, fundamentals, search, vectors* --- > [!info] Quick Definition > An **embedding** is a numerical representation of an item, such as a text, an image or a user, as a vector (a list of numbers). It is designed so that items with similar meaning sit close to each other in that numerical space. ## What is an embedding? An embedding turns something complex into a list of numbers. The list is learned from data: a model adjusts its values so that items appearing in similar contexts receive nearby vectors. It is the idea John Rupert Firth summed up in 1957: a word is characterized by the company it keeps. Illustrative example: "chair" and "stool" would end up closer to each other than "chair" and "storm". Not because someone tagged them by hand, but because they appear in similar contexts in the training data. Embeddings are not meaning itself. They are a numerical approximation that is useful for comparison. When that approximation does not match what the user wants, the comparison fails. ## How are they generated, and how big are they? An embedding is produced by an **embedding model**: you give it a text (or an image) and it returns the vector. A little history is useful here: - **Word embeddings.** In 2013, a Google team led by Tomas Mikolov created word2vec, which popularized the technique. Each word has a single vector, regardless of the sentence it appears in. - **Contextual embeddings.** Later models such as ELMo and BERT produce a different vector for each occurrence of a word, so "bank" in "river bank" and "bank" in "bank account" do not have to end up the same. - **Other types.** There are also embeddings for images, knowledge graphs and user interactions, which are used in recommendation. Size depends on the model. According to OpenAI's documentation, `text-embedding-3-small` returns 1536 numbers by default and `text-embedding-3-large` returns 3072. More numbers do not mean "better" on their own: they mean more storage and more computation for every comparison. ## How do you compare two embeddings? The most common measure is **cosine similarity**, which compares the direction of two vectors regardless of their length. Its value ranges from -1 to 1, narrowing to 0 to 1 when the components cannot be negative. Near 1, the vectors point in similar directions; at 0 there is no relation; at -1 they point in opposite directions. OpenAI recommends cosine similarity for its models. Illustrative example with just two dimensions (real ones have hundreds or thousands). With A = (3, 4), B = (4, 3) and C = (-4, 3): | Pair | Cosine similarity | Reading | |---|---|---| | A and B | 0.96 | Almost the same direction: very similar | | A and C | 0 | Perpendicular: unrelated | This enables one operation that powers many features: given the vector of a query, find the closest vectors in a collection. That is the core of [[semantic-search|semantic search]]. When the collection is large, the vectors are stored in a [[vector-databases|vector database]]. ## Why does this matter to a designer? Because embeddings explain behavior users notice: - **"Similar" results that are not identical.** A search by meaning can return a result that does not contain the typed word. That can be useful or confusing, depending on the case. - **Automatic grouping.** If the product clusters tickets, notes or designs by similarity, the quality of the groups depends on the embeddings. - **Retrieval for [[rag|RAG]].** The document fragments that reach the model are often chosen by embedding similarity. If retrieval fails, the answer fails too. - **Recommendations.** "You might like" often comes from comparing vectors of items or users. - **Inherited bias.** Embeddings reflect associations in their training data. Bolukbasi and colleagues (2016) showed that word embeddings trained on Google News reproduced gender stereotypes, and proposed a method to reduce them. More in [[model-bias|bias in models]]. ## When NOT to use them? When search needs exact matches: an order number, a product code, a proper name that must be found exactly as typed. A keyword search or a traditional index is usually more reliable there. Many products combine both: exact search for identifiers and embeddings for natural language. They also do not help when the collection is tiny (say, ten menu options): a simple filter is enough and easier to explain to the user. ## Example Illustrative example: a library of design components has 400 entries with descriptions. The team generates one embedding per entry. When someone types "button to confirm a destructive action", the system returns "confirmation dialog" and "danger button" even though neither contains those exact words. It also returns an "alert" component the designer did not expect. That result is not a system bug; it is a signal. What the designer can do: 1. Review the component descriptions so they say what each one is for, not just how it looks. 2. Show a short explanation next to each result of why it appeared. 3. Build a list of 20 or 30 real queries with the expected answer and rerun it whenever the model or the descriptions change. ## Common mistakes - Treating an embedding as the exact meaning of the text. - Comparing vectors produced by different models. Each model has its own space and often a different size (1536 versus 3072 numbers, for example). - Forgetting to store the original text and a reference to its source. - Evaluating search with two queries instead of a representative set. - Using embeddings for identifiers that should be searched exactly. - Ignoring the model's biases in features that classify or recommend people. ## Sources - [Embedding (machine learning), Wikipedia](https://en.wikipedia.org/wiki/Embedding_(machine_learning)) - [Word embedding, Wikipedia](https://en.wikipedia.org/wiki/Word_embedding) - [Cosine similarity, Wikipedia](https://en.wikipedia.org/wiki/Cosine_similarity) - [OpenAI: embeddings guide](https://developers.openai.com/api/docs/guides/embeddings) - [Bolukbasi et al. (2016), Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://arxiv.org/abs/1607.06520) --- Source: https://www.fernandoux.com/en/wiki/ai/embeddings/