The Ultimate Guide to Embeddings and Vector Databases

These 2 Tools Are Behind Any Advanced AI Application

Sep 25, 2024

Artificial intelligence is on the rise and two concepts are becoming increasingly crucial for anyone building AI products: embeddings and vector databases.

These powerful tools are the secret sauce behind many advanced AI applications, from chatbots with long-term memory to semantic search engines that can understand the meaning behind your queries.

But what exactly are they, and how can you harness their potential?

What Are Embeddings?

They’re not just random numbers; embeddings are numerical representations that models learn through extensive training.

These vectors capture the relationships within the data by analyzing how often certain patterns co-occur.

Suppose you’re trying to organize a massive library. Instead of arranging books alphabetically or by genre, what if you could somehow map out how similar or different each book is to every other book in the collection?
That’s essentially what embeddings do but in a mind-bogglingly complex way.

Let’s break it down with a simple example.

We know that “dog” and “puppy” often appear in similar contexts, right?

In an embedding, these words would be represented by vectors that are close together in the mathematical space.

While we might visualize this as a simple 2D graph, real embeddings typically use hundreds of dimensions to capture the rich, multifaceted relationships between words or concepts. It’s like each word gets its unique address in a hyper-dimensional city!

But words aren’t the only things we can embed. Images, too, can be turned into these numerical representations.

In fact, it’s how Google pulls off those impressive “find similar images” searches. The image gets broken down into a vector, and the search engine looks for other images with similar vector “addresses.”

Vector Databases: Where Embeddings Live

Once you’ve created these embeddings, you need somewhere to store them. That’s where vector databases come into play.

These are specialized databases designed to house and quickly retrieve these high-dimensional vectors.

Vector databases are incredibly versatile. They can be used for:

Searching: Ranking results based on how closely they match a query.
Clustering: Grouping similar items.
Recommendations: Suggest items related to what you’re currently looking at.
Classification: Categorizing text based on its most similar known label.

For this article, we’ll focus on searching, as it’s one of the most common and powerful applications.

#Ad: I would love it if you check out my eBooks to support me:

🔗 Top 50+ ChatGPT Personas for Custom Instructions

Also, get free data science & AI eBooks: https://codewarepam.gumroad.com/

Getting Started with OpenAI’s Embeddings

Now that we understand the theory, let’s roll up our sleeves and create some embeddings! OpenAI provides a fantastic API for generating embeddings. While they don’t offer storage solutions, we’ll cover that in a bit.

First things first, head over to OpenAI’s website and create an account (or log in if you already have one).
Once you’re in, navigate to the API section. Here, you’ll find all the documentation you need to start creating embeddings.

Note: Keep in mind, that the OpenAI API has rate limits and usage costs, so be sure to check your plan details on OpenAI’s pricing page to avoid unexpected charges.

Next, to interact with the API, we’ll use a handy tool called Postman. It’s a user-friendly platform for making API requests without having to write complex code.

You can download it for free from their website.
Once you have Postman set up, create a new workspace for your OpenAI experiments.
Then, set up a new POST request to the OpenAI embeddings endpoint: https://api.openai.com/v1/embeddings
You’ll need to authenticate your requests using an API key from OpenAI.
Generate one from your account dashboard and add it to your Postman request as a bearer token.

Now comes the fun part — creating your first embedding!

In the body of your Postman request, you’ll need to specify two things:

The model: We’ll use “text-embedding-ada-002”, which is OpenAI’s recommended (and most cost-effective) embedding model.
The input: This is the text you want to convert into an embedding.

Let’s start simple. Use this JSON in your request body:

{
 "model": "text-embedding-ada-002",
 "input": "Hello, world!"
}

Hit send, and voilà! You’ve just created your first embedding. The response will include a long list of numbers — that’s your embedding vector.

Exploring Different Types of Embeddings

Now that you’ve got the hang of it, let’s explore some different types of embeddings:

Single-word embeddings: Perfect for simple searches or classifications.
Multi-word embeddings: Great for capturing the nuance of short phrases or sentences.
Paragraph or document embeddings: Ideal for summarizing larger chunks of text.

OpenAI’s Ada model can handle up to 8,000 tokens (which typically equates to roughly 6,000 to 8,000 words, depending on the text structure). That’s about 10 pages of text! This makes it possible to embed entire documents or long articles in one go.

Storing Embeddings: Enter Vector Databases

Creating embeddings is only half the battle. To make them truly useful, we need to store them somewhere. This is where vector databases come in handy.

For this tutorial, we’ll use SingleStore, a cloud-based database that supports vector storage and searching.

Head over to SingleStore’s website and create a free account.

Once you’re set-up, create a new workspace and database.
Then, using their SQL editor, we can create a table to store our embeddings:

CREATE TABLE IF NOT EXISTS my_vector_table (
  text TEXT,
  vector ARRAY
);

This simple table will store the original text and its corresponding embedding vector. In SingleStore, we store vectors as arrays instead of blobs, making retrieval and similarity search faster and more efficient.

To insert an embedding, we’ll use an SQL INSERT statement. Here’s an example:

INSERT INTO my_vector_table (text, vector)
VALUES (
 'Hello, world!',
 JSON_ARRAY_PACK('[0.1, 0.2, 0.3, …]') - Replace with your actual embedding
);

Searching Vector Databases

Now for the exciting part — searching our vector database! The process goes like this:

Create an embedding for your search term.
Use that embedding to search the database.
Rank the results based on similarity.

Here’s a sample SQL query to search our vector database:

SELECT 
  text, 
  COSINE_SIMILARITY(vector, '[0.1, 0.2, 0.3, ...]') AS score
FROM my_vector_table
ORDER BY score DESC
LIMIT 5;

Replace the numbers in JSON_ARRAY_PACK with the actual embedding of your search term.

In the above query, we calculate the cosine similarity between your search term and stored vectors, returning the top 5 results with the highest similarity score.

Wrapping Up

Embeddings and vector databases are powerful tools that open up a world of possibilities in AI and machine learning. From enhancing search capabilities to powering recommendation systems, these technologies are at the heart of many cutting-edge applications.

So go forth and experiment! Create embeddings, build your vector database, and see what amazing things you can create.

If you found this article useful, consider ❤️ liking it, and you are also welcome to support me by tipping your desired amount here.

Connect: LinkedIn | Gumroad Shop | Medium | GitHub

Subscribe: Substack Newsletter | Appreciation Tip: Support

Your Data Guide

Discussion about this post