Using AI to Find Similar Textbook Questions

Recently, someone approached me with an interesting problem.

A student working through a mathematics and statistics textbook would sometimes get stuck on an exercise. Once they understood the concept, the most useful next step was often to practise another question testing the same underlying idea.

The challenge was finding one.

The textbook was more than 900 pages long, and manually searching through hundreds of exercises for "another question like this one" was slow enough that, in practice, it often didn't happen.

That led to a simple question:

Could AI help find genuinely similar practice questions from the textbook itself — without generating new questions or providing answers?

That became the starting point for AI Question Finder.

The idea

AI Question Finder is deliberately a retrieval system rather than a chatbot.

A student pastes in a question they are struggling with, and the system searches the textbook's existing exercise bank to find questions that test similar concepts.

• No generated questions.
• No AI-written answers.
• No invented textbook content.

The textbook remains the source of truth. AI simply helps the student navigate it more effectively.

First challenge: extracting the questions

Before any AI could be useful, the textbook had to be turned into a reliable question bank.

That turned out to be more interesting than expected.

The PDF contains numbered exercises, multi-part questions, section numbers, references to other exercises and page breaks that sometimes split a question across multiple pages.

A simple regex over the PDF wasn't going to be reliable enough.

I built a two-pass extraction pipeline in Python that processes the textbook page by page and identifies real exercises while preserving:

• chapter
• section
• page number
• question number
• multi-part questions such as a, b and c

It also distinguishes genuine exercises from things that only look like question numbers, such as section headings and references to earlier exercises.

The result was:

1,555 exercises extracted from a 939-page textbook.

Only 12 questions — around 0.8% — were flagged for manual review.

That gave the retrieval system a solid foundation.

What does "similar" actually mean?

This was the most interesting part of the problem.

Traditional keyword search is not particularly good at this use case. Two mathematics questions can use completely different wording while requiring exactly the same concept and reasoning.

So each extracted question is enriched using an LLM with attributes such as:

• topic
• subtopic
• required skills
• difficulty
• reasoning steps

Each question is then converted into a vector embedding representing its mathematical meaning and stored in Chroma.

When a student submits a question, the same process happens in reverse. The query is embedded, matched against the exercise bank, and the strongest candidates are reranked using signals such as topic, skills and difficulty.

The goal is not:

"Find another question containing the same words."

It is:

"Find another question that requires the student to think in the same way."

Testing it against the real textbook

I wanted to make sure the system wasn't just producing results that looked plausible. It needed to be tested against known examples.

One worked example involved a question about three radar sets. When that question was submitted, the system returned the textbook's actual Exercise 2.112 as the top match, including the correct chapter, page and all of its sub-parts.

The similarity score was around 63%.

I also created a hand-verified evaluation set using questions from a real class test. For questions where equivalent textbook exercises had been confirmed, the system achieved:

9/9 recall@5.

Every confirmed matching exercise appeared within the first five results.

For me, that validation was much more valuable than simply looking at a few impressive demos.

The technology

The implementation is intentionally straightforward:

• Python + FastAPI + LangChain for ingestion and retrieval
• Chroma for the vector database
• Next.js + TypeScript for the user interface
• OpenAI for embeddings and classification

The interesting part was not adding more AI. It was getting the extraction, retrieval and ranking reliable enough that the user could trust the result.

The cost surprised me

Because the complete dataset already existed, I was also able to measure the actual OpenAI API cost.

Using gpt-4o-mini for enrichment and classification and text-embedding-3-small for embeddings:

• One-time ingestion of all 1,555 exercises: approximately $0.13
• Cost per search: approximately $0.00007

At that rate, around $10 of API usage would support roughly 140,000 searches.

For this kind of application, the AI inference cost is almost insignificant.

What I deliberately left out

It would be easy to keep adding features, but I wanted the first version to stay focused and trustworthy.

So there are:

• no textbook answers displayed
• no AI-generated questions
• no AI-generated solutions
• no student progress tracking

The purpose is very specific:

Take a difficult question and help the student find the best existing question to practise next.

What I took away from building it

What I found interesting about this project is that AI is doing very little "creation."

Instead, it is helping a user navigate a large body of trusted information. The source content remains authoritative. The model is being used to understand meaning, establish relationships and retrieve the right information at the right moment.

Sometimes the better question is:

Can AI help us find the right piece of trusted content, exactly when we need it?

---

That is what this project became for me.