Building Neo4j-Powered Applications with LLMs E-Book

Ravindranatha Anthapu

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Veröffentlichungsjahr: 2025

Beschreibung

Embark on an expert-led journey into building LLM-powered applications using Retrieval-Augmented Generation (RAG) and Neo4j knowledge graphs. Written by Ravindranatha Anthapu, Principal Consultant at Neo4j, and Siddhant Agrawal, a Google Developer Expert in GenAI, this comprehensive guide is your starting point for exploring alternatives to LangChain, covering frameworks such as Haystack, Spring AI, and LangChain4j.
As LLMs (large language models) reshape how businesses interact with customers, this book helps you develop intelligent applications using RAG architecture and knowledge graphs, with a strong focus on overcoming one of AI’s most persistent challenges—mitigating hallucinations. You'll learn how to model and construct Neo4j knowledge graphs with Cypher to enhance the accuracy and relevance of LLM responses.
Through real-world use cases like vector-powered search and personalized recommendations, the authors help you build hands-on experience with Neo4j GenAI integrations across Haystack and Spring AI. With access to a companion GitHub repository, you’ll work through code-heavy examples to confidently build and deploy GenAI apps on Google Cloud.
By the end of this book, you’ll have the skills to ground LLMs with RAG and Neo4j, optimize graph performance, and strategically select the right cloud platform for your GenAI applications.

Leseprobe

Building Neo4j-Powered Applications with LLMs

Create LLM-driven search and recommendations applications with Haystack, LangChain4j, and Spring AI

Ravindranatha Anthapu

Siddhant Agarwal

Building Neo4j-Powered Applications with LLMs

Copyright © 2025 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Portfolio Director: Gebin George

Relationship Lead: Sonia Chauhan

Project Manager: Prajakta Naik

Content Engineer: Vandita Grover

Technical Editor: Rahul Limbachiya

Copy Editor: Safis Editing

Indexer: Tejal Soni

Proofreader: Vandita Grover

Production Designer: Alishon Falcon

Growth Lead: Nimisha Dua

First published: June 2025

Production reference: 1230525

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83620-623-1

www.packtpub.com

I would like to dedicate this book to my lovely wife, Sreevani Rajuru, for all her encouragement and patience during this journey.

– Ravi Anthapu

I would like to first and foremost thank my loving and patient wife, Rashi, for her unwavering support, encouragement, and endless patience throughout the long journey of writing this book. To my wonderful parents—thank you for your constant belief in me and for being my lifelong pillars of strength. I’m also deeply grateful to my incredible colleagues whose camaraderie, insights, and inspiration pushed me to keep going. A heartfelt thank you to the Neo4j team and community for fostering an environment of innovation and collaboration—your support made this book possible.

– Sid Agarwal

Forewords

For much of my career, AI has been in a kind of nuclear winter, a stasis from which we were told amazing things would emerge in the next 10 years. That winter began to ever-so-slightly thaw with practical applications of machine learning over the last decade but with truly amazing things still a perpetual 10 years away.

And then ChatGPT changed everything.

Today, AI is no longer the preserve of academic journals and ambitious conference talks. It has taken center stage in the IT world and is reshaping entire industries and economies. For technology professionals, it has redefined what it means to build software in the modern era.

The shift has been swift and dizzying. We have all marveled at GenAI-powered chatbots, but amid the excitement and innovation, a serious question has lingered: how can we use AI dependably? Dependability is an important characteristic of systems. It encompasses aspects such as availability and reliability but of particular importance in a world built on AI, it also covers safety, security, and trustworthiness.

It is in the ability to help systems developers make AI trustworthy that Sid and Ravi as authors, make their mark.

This book is a thoughtful, well-written guide from authors with real experience in contemporary graph and AI systems. The authors know the terrain, including theory and architecture as well as code and the deployment pipeline. They have accumulated wisdom that is very worthy of sharing.

The book builds on a classic RAG approach, where retrieval from external data sources meets generation to produce grounded, useful responses. From there, the reader is gently guided into the more sophisticated and powerful world of GraphRAG, where Neo4j and knowledge graphs play a central role in helping foundational models to produce trustworthy results. Along the way, readers are acquainted with examples using modern AI middleware, such as Neo4j’s GenAI framework, with a focus on Haystack, LangChain4j, and Spring AI. This middleware helps us to rapidly build and deploy (including on the cloud) coherent, dependable systems supported by knowledge graphs.

The authors write with clarity and authority that reflects their experience. They understand both how to make knowledge graphs and GenAI systems work and how to explain them in a way that’s accessible and engaging. But what makes this book particularly outstanding is its developer-first sensibility. There’s no hand-waving or marketing speak—just clear, pragmatic advice, and a collection of crisply written technical examples. It is the sort of material that’s empowering for developers who are tasked with putting dependable AI systems into production.

Dr. Jim Webber

Chief Scientist, Neo4j

My journey with Neo4j began in 2016 during my master’s thesis, where I used it to traverse citation graphs between scientific papers and patents to recommend cross-collection references. Coming from a background of natural language processing, and having worked mostly with unstructured data, I was struck by the clarity of graph traversal and the elegance of the Cypher query language.

Two years later, in 2018, deepset was founded, and the foundations of the open source AI framework Haystack were soon built. Designed for building modern, compound AI systems with a focus on production readiness, the framework quickly gained a vibrant community and with it a growing variety of use cases and deployed solutions. Fast forward to today, and the integration of Haystack and Neo4j feels not just natural but like the next logical step in building smarter, context-aware AI applications.

As one of the maintainers of Haystack, I am particularly excited about this book. It doesn’t just present a theoretical case for using knowledge graphs with LLMs. It is a hands-on, pragmatic guide to building RAG-based applications with Neo4j’s powerful graph model and Haystack’s flexible pipeline architecture with its highly customizable components. As Sergey Bondarenco, who worked on the integration, put it well: “Haystack pipelines are also graphs.” Each component is a node, each connection an edge, you can have branches and loops, and when you run a pipeline, information flows from one component to the next. With Neo4j’s integration, you can now filter metadata through Cypher, combine vector and structured search in a single query, and enhance context windows of LLMs with structured information.

In a field moving as fast as GenAI, it is refreshing to see a book that is grounded in well-established, robust technology. Ravi and Sid have written a guide that embraces the best of both worlds: the structure and interpretability of graphs, and the flexibility and reasoning capabilities of LLMs. Whether you are just starting with RAG or looking to level up your graph-driven GenAI applications, this book will meet you where you are and take you further.

Dr. Julian Risch

Team Lead (Open Source Engineering), deepset

Contributors

About the authors

Ravindranatha Anthapu has more than 25 years of experience in working with W3C standards and building cutting-edge technologies, including integrating speech into mobile applications in the 2000s. He is a technology enthusiast who has worked on many projects, from operating system device drivers to writing compilers for C language and modern web technologies, transitioning seamlessly and bringing experience from each of these domains and technologies to deliver successful solutions today. As a principal consultant at Neo4j today, Ravindranatha works with large enterprise customers to make sure they are able to leverage graph technologies effectively across various domains.

Siddhant Agarwal is a seasoned DevRel professional with over a decade of experience in cultivating innovation and scaling developer ecosystems globally. Currently leading developer relations across APAC at Neo4j and recognized as a Google Developer Expert in GenAI, Sid transforms local developer initiatives into global success stories with his signature “Local to Global” approach. Previously working at Google managing flagship developer programs, he has shared his technical expertise at diverse forums worldwide, fueling inspiration and innovation. Learn more about him and his work, visit, meetsid.dev (https://meetsid.dev/).

About the reviewers

Fernando Aguilar Islas is a senior solutions consultant at Enterprise Knowledge, leading the design and implementation of AI and graph-based solutions for global organizations. He specializes in semantic solutions, LLMs, graph data science, knowledge graphs, and machine learning. His work spans diverse use cases, including identity resolution with graphs, AI-augmented human-in-the-loop frameworks, and graph-based recommenders. Fernando holds an MS in applied statistics from Penn State and is a certified Neo4j Professional and AWS Solutions Architect Associate. He is passionate about building intelligent systems that drive measurable business impact through advanced analytics, semantics, and graph technologies.

Manisha Mittal is a machine learning engineer with nearly a decade of experience in delivering enterprise-grade solutions at leading technology companies such as SAP and Visa. She specializes in GenAI-based RAG (embeddings and GraphRAG) applications, agentic workflows, and evaluation frameworks, along with in-depth knowledge of LLMs. She has deep expertise across a wide tech stack, including Java, Python, SQL, Hive, Neo4j, React, Node.js, Airflow, and Hadoop. Manisha holds a B.Tech. degree in information technology from the Indian Institute of Information Technology, Allahabad, India, with a strong foundation in scalable application development and AI-driven systems. 

Mihir Arya is a UCLA mathematics graduate currently working at Visa as a senior software engineer. His team works heavily in the GenAI space, in the domains of building agentic user applications and GenAI evaluation frameworks, in addition to building and promoting knowledge graphs as a revolutionary alternative to traditional vector RAG among internal developers.

Acknowledgement

The authors are grateful to the grouplens team and the Kaggle team for making available the Movies dataset, a derivative of the Movie Lens Datasets.

(F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. https://doi.org/10.1145/2827872).

The authors are grateful to H&M Group and the Kaggle team for making available the H&M Personalized Fashion Recommendations dataset.

(Carlos García Ling, ElizabethHMGroup, FridaRim, inversion, Jaime Ferrando, Maggie, neuraloverflow, and xlsrln. H&M Personalized Fashion Recommendations. 2022. Kaggle. https://kaggle.com/competitions/h-and-m-personalized-fashion-recommendations)

Join our communities on Discord and Reddit

Have questions about the book or want to contribute to discussions on Generative AI and LLMs?

Join our Discord server at https://packt.link/4Bbd9 and our Reddit channel at https://packt.link/wcYOQ to connect, share, and collaborate with like-minded enthusiasts.

Part 1

Introducing RAG and Knowledge Graphs for LLM Grounding

This first part of the book sets the stage for building grounded, context-aware AI applications. We will start by introducing the fundamentals of Large Language Models (LLMs), the challenges they face around factuality, and how Retrieval-Augmented Generation (RAG) helps address those limitations. Next, we break down RAG architectures with practical insights and implementation guidance. We conclude by establishing a foundational understanding of knowledge graphs—highlighting how Neo4j enables structured, semantically rich representations that enhance the grounding and reasoning capabilities of LLMs.

This part of the book includes the following chapters:

Chapter 1, Introducing LLMs, RAGs, and Neo4j Knowledge GraphsChapter 2, Demystifying RAGChapter 3,Building a Foundational Understanding of Knowledge Graph for Intelligent Applications

Stay tuned

To keep up with the latest developments in the fields of Generative AI and LLMs, subscribe to our weekly newsletter, AI_Distilled, at https://packt.link/Q5UyU.

1

Introducing LLMs, RAGs, and Neo4j Knowledge Graphs

Artificial Intelligence (AI) is evolving beyond niche and specialized fields to become more accessible and able to assist with day-to-day tasks. One of the best examples is the explosive advent of Generative AI (GenAI). In the last few years, GenAI has created a lot of excitement both for technology builders and regular users with its ease of use and ability to understand and answer questions the way humans can. The breakthroughs in Large Language Models (LLMs) have propelled GenAI to the forefront. This has opened up a lot of opportunities for businesses to change how they interact with their customers. Customers can ask a question in natural language and get an answer without needing a human to be available to understand the question or understand the data to extract intelligence from it. While GenAI has taken big strides in different fields with different modalities, such as text, audio, and video, our focus throughout this book remains on LLMs and their applications in business and industry use cases.

In this chapter, we will take a look at GenAI through the lens of LLMs, its impact, pitfalls, and ethical concerns. To set the stage for this book, we will briefly introduce techniques that can augment LLMs to make them more effective.

In this chapter, we are going to cover the following main topics:

Outlining the evolution of GenAI through the lens of LLMsUnderstanding the importance of RAGs and knowledge graphs in LLMsIntroducing Neo4j knowledge graphs

Outlining the evolution of GenAI through the lens of LLMs

In late 2022, OpenAI took the world by storm by releasing an AI engine called ChatGPT that could understand language like humans and interact with users in natural language. This was the best representation of GenAI in a long time. AI concepts started as rules-based systems and evolved into machine learning algorithms in the ‘90s. With the rise of deep learning and LLMs, the concept of GenAI became more popular. These AI systems could generate new content after being trained using existing content. OpenAI’s GPT-3 LLM model was one of the first LLMs that captured the interest of the masses. GenAI can be used to get answers in a manner that feels like human interaction and it can also be used to generate images by providing a text description, describe an image as text content, generate videos using text content, and many other things. It can enhance creativity, accelerate research and development, enable a simple understanding of complex concepts, and improve personalization.

The evolution of LLMs is at the heart of GenAI’s popularity. Let’s take a look at LLMs and how they have propelled GenAI.

Introducing LLMs

An LLM is a machine learning model that is built for natural language processing and can understand language constructs and generate content in that language based on the training.

Before the popularity of GPT-3, a considerable amount of research had been conducted on LLMs fover several years. Some of the notable works that pioneered LLMs include Google’s Bidirectional Encoder Representations from Transformers (BERT (https://github.com/google-research/bert)) and Generative Pre-trained Transformer (GPT) from OpenAI. LLM training requires a lot of parameters and computing power.

At their core, LLMs are a type of Recurrent Neural Network (RNN) architecture. Traditional RNNs struggle with long-term dependencies in sequential data. To address this, LLMs often leverage architectures such as Long Short-Term Memory (LSTM) networks or transformers. These architectures allow the model to learn complex relationships between words, even words that are separated by large distances within the training text.

Here is a simple illustration of a basic LLM architecture:

Figure 1.1 — Flowchart explaining a basic LLM architecture

Let’s dissect this architecture

Input layer: This layer receives the initial text prompt or sequenceEmbedding layer: Words in the input sequence are converted into numerical vectors, capturing their semantic meaningEncoder: This is a multi-layered RNN (e.g., LSTM) or transformer that processes the sequence of embedded words, capturing contextual informationDecoder: The decoder utilizes the encoded representation to generate the output sequence one word at a time

You can read more about LLMs in this paper: https://arxiv.org/pdf/2307.06435.

Building an LLM requires a lot of effort and resources. Let’s look at the number of parameters used by OpenAI to train each GPT model:

GPT-1: This is the first model and used 117 million parameters.GPT-2: This model used 1.5 billion parameters to train.GPT-3: This model was the first general-purpose model released. 175 billion parameters were used to train this model.GPT-4 series: This is the latest model released by OpenAI. 170 trillion parameters were used to train this model.

These training figures demonstrate that with each new version, the number of parameters increased by several orders of magnitude. This means more and more computing power is needed to train these models. Similar training numbers can be observed for other LLM models too.

While GenAI is a great technology, there are pitfalls as well as legal and ethical concerns about the application of this technology. We will take a look at them next.

Understanding GenAI’s pitfalls and ethical concerns

While LLMs are great at summarizing, generating context, and other use cases, they still do not understand the language per se. They recognize patterns based on the training text to generate new text. They also don’t understand facts or understand emotions or ethics. They are simply predicting the next token and generating text. Because of these pitfalls, content generated by GenAI can have huge consequences.

To understand and address these aspects, we need to first identify any harmful or inaccurate content that is being generated and address it either by retraining the model or adding separate checks and balances to make sure this content is not used as output.

For example, there have been recent cases about using LLMs to generate legal briefs, where LLMs have created non-existent cases and generated a legal brief based on these cases. While technically it might have generated a solution that is requested, this is legally not correct. There have also been cases where LLMs are used to generate offensive images and videos and shared on the internet. Since it is difficult to identify content generated by AI, it is easy to be fooled by this content. This is neither socially, legally, or ethically acceptable. There are quite a few examples where LLMs simply make up facts.

This tutorial on the Microsoft site (https://learn.microsoft.com/en-us/training/modules/responsible-ai-studio/) provides a detailed explanation of these concerns and how we can identify them.

Retrieval-Augmented Generation (RAG) and knowledge graphs together can help address these issues, which we discuss next.

Understanding the importance of RAGs and knowledge graphs in LLMs

To address the pitfalls of GenAI, we can either fine-tune the model or ground the responses using other sources.

Fine-tuning involves training an existing model with additional information, which can result in high-quality responses. But this can be a complex and time-consuming process.

The RAG approach involves providing extra information when we are asking the LLM a question.

With this approach, you can integrate knowledge repositories into the generative process. In this scenario, LLM can leverage the extra information retrieved from other sources and tune the response to match the information provided, thus grounding the results.

These repositories and sources can include the following:

Publicly available structured datasets (e.g., scientific databases such as PubMed or publicly accessible encyclopedic resources such as Wikipedia)Enterprise knowledge bases (e.g., internal company documentation, product catalogs, or compliance-related content with strict privacy and security requirements)Domain-specific sources (e.g., legal case records, medical guidelines, or technical manuals tailored to specific industries)

By incorporating relevant information from these repositories and sources, RAG empowers LLMs to generate output that is not only factually accurate but also contextually aligned with the task at hand. Unlike the static knowledge encoded in the LLM’s training data, these additional data sources allow real-time retrieval of up-to-date and specialized information, addressing challenges such as data freshness, accuracy, and specificity. We will cover RAG in detail in Chapter 2.

Another source of information to enable RAG is knowledge graphs. Let’s briefly talk about them and their role in the LLM landscape.

The role of knowledge graphs in LLMs

Knowledge graphs play a huge role in generating creative and contextually rich content for LLMs. They provide a structured, interconnected foundation and make information retrieval more relevant and insightful by grounding the AI results in a complex and multi-layered understanding of the data.

Representing the data as a graph opens up more avenues to understand the data. At the same time, a knowledge graph cannot be a static entity that represents data in only one dimension that’s fixed. Its true power lies in its ability to be dynamic and multi-dimensional. It can capture temporal, spatial, or contextual information in real time through live data feeds.

Apart from being an important tool for storing information, knowledge graphs are the backbone of intelligent, context-aware AI.

There are several reasons why knowledge graphs are essential for GenAI:

Enhanced contextual understanding: Knowledge graphs allow GenAI systems to retrieve relevant information based on relationships, not just isolated facts. For example, in healthcare, a knowledge graph could link symptoms, diseases, and treatments, enabling GenAI to suggest more accurate diagnostic insights based on interconnected medical knowledge.Efficient data retrieval: Unlike traditional databases, knowledge graphs allow multi-hop reasoning, from which GenAI can draw insights across several degrees of separation. This is invaluable in fields such as finance, where GenAI can use knowledge graphs to reveal hidden relationships between entities such as customers, transactions, and market trends.Integration of vector embeddings: When combined with vector embeddings, knowledge graphs enable GenAI to understand and respond to more nuanced queries. Vector embeddings capture semantic similarities between data points, which knowledge graphs then contextualize, creating a powerful blend of accuracy and relevance in responses.Real-world impact: Major organizations are already harnessing the power of knowledge graphs to enhance GenAI applications. For instance, companies in e-commerce use knowledge graphs to provide product recommendations that are not just relevant but contextually rich, drawing from diverse data sources such as customer reviews, purchase history, and product features.

By integrating knowledge graphs, GenAI models transcend traditional data limitations, helping to create smarter, more reliable applications across different fields.

Let’s now talk about Neo4j knowledge graphs.

Introducing Neo4j knowledge graphs

A knowledge graph is dynamic and continues to evolve based on how data and relationships within the data evolve with time.

Neo4j is a database that excels with its ability to store data in graphs. For example, in a store, most products are laid out in a certain grouping and stay in those groups. But there is an exception to this arrangement. When a store wants to promote some products, they are placed at the front of the store. This kind of flexible thought process should be adapted for our knowledge graph implementation. As the semantics of data evolves the knowledge graph should be able to capture this change.

Neo4j, with its multiple labels for nodes and its optional schema approach, makes it easy to keep our graph relevant by helping us to persist (retain) our understanding of data as an extra label on the node, or a specific relationship that provides more relevant context between the nodes. We will take a deeper look at how we can build a Neo4j knowledge graph from the ground up in the upcoming chapters.

For now, let’s see how a Neo4j knowledge graph works to enhance an LLM’s response.

Using Neo4j knowledge graphs with LLMs

Suppose there is an LLM-based chatbot integrated with a Neo4j knowledge graph. This GenAI chatbot is designed to answer medical queries. Figure 1.2 illustrates how a Neo4j knowledge graph can enhance this chatbot’s medical reasoning by linking structured patient symptom records with unstructured insights from medical research papers and clinical trials.

The unstructured text undergoes embedding-based processing using models from providers such as Ollama, OpenAI, and Hugging Face, followed by Named Entity Recognition (NER), to extract key entities such as symptoms and treatments. This data is integrated into a Neo4j knowledge graph, where documents mention symptoms and treatments, patients show symptoms, and symptoms are linked to potential treatments. This enables multi-hop reasoning, allowing a chatbot to efficiently answer complex queries such as the following:

Which patients are showing symptoms similar to flu and also showed symptoms of COVID-19 in the past?

Figure 1.2 — Neo4j knowledge graph driven Gen-AI for healthcare

To retrieve the result of this query, a multi-hop knowledge graph query path (Figure 1.2) will be followed in this order:

Retrieve symptoms linked to flu from research documents.Identify patients currently showing those symptoms.Cross-reference past patient records for COVID-19 symptoms.Return patients who match both conditions with supporting document sources.

With this approach, the LLM response can be grounded to generate factually correct, relevant, and up-to-date results to support medical decision-making.

A similar approach can be used to augment LLMs that support other applications.

We have now looked at how knowledge graphs enhance GenAI’s ability to provide contextually rich, accurate insights. But how does this transformative power translate into concrete benefits in real life? We will continue this journey in the rest of the book.

Summary

In this chapter, we discussed the evolution of GenAI in the context of LLMs. We also looked at how RAG and knowledge graphs are key enablers of this transformation and help provide structure and context, improving an LLM’s accuracy and reasoning.

Looking ahead, the next chapter dives deep into RAG — a technique that significantly enhances GenAI’s accuracy by grounding responses in retrieved, verified information.