Databases on foojay.io - Friends of OpenJDK

Getting Started with Exposed: Kotlin ORM Made Simple

Mon, 06 Jul 2026 03:39:00 +0000

Introduction

For quite some time, I have been a huge fan of and fascinated by JetBrains products, tools, and libraries because of their masterful craftsmanship in product creation and their pristine focus on building high-quality developer tools.

Even more excitingly, JetBrains Java Annotated Monthly newsletters have featured most of the technical articles I wrote for Foojay on topics such as Java, Spring, Spring Boot 4, and OpenRewrite.

Recently, one Kotlin Domain-Specific Language (DSL) library caught my attention. I immediately tried converting my existing Spring Boot 4 application from Java to Kotlin using Exposed, an ORM framework for Kotlin. Kotlin SQL Libary

Aggregation Optimization in MongoDB: Unnecessary Unwinds (Part 2)

Thu, 25 Jun 2026 10:01:00 +0000

And why MongoDB might be a better relational database than you ever realized.

Design reviews are one-on-one meetings where MongoDB experts deliver advice on data modeling best practices and application design challenges. In this series, we are going to explore common real-life scenarios where design reviews helped developers achieve meaningful success with MongoDB.

This article was written by Graeme Robinson. Find him on LinkedIn .

In Part 1 of this series , we described a use case based on a recent design review I conducted with a team at a MongoDB customer. The team in question was new to MongoDB, and the approach they had taken to both modeling their data and then subsequently querying it was very “RDBMS-like.” As a result, query performance was significantly slower than their SLA called for.

BoxLang 1.14.0 : Query Transformers - Take Full Control of Your Query Results

Wed, 24 Jun 2026 10:46:13 +0000

BoxLang 1.14.0 ships a lot of exciting features – Dynamic Sets, Ranges, Inner Classes, JSONPath navigation – but one quietly powerful addition will change the way you think about every database call in your application: Query Transformers, and this is just the start, we have plans for a whole lot more cool query features.

If you have ever executed a query and then immediately written a loop to reshape the result into what you actually needed, this feature is for you.

Building an AI-Powered Operations Assistant with Spring AI and MongoDB Atlas — Part 2: Conversational Memory

Wed, 10 Jun 2026 19:23:46 +0000

This is the second article in a three-part series. Part 1 covered the RAG foundation — loading runbooks into a vector store and wiring them to a language model. Part 3 will introduce stateful workflow checkpointing with pause and resume.

The Problem with Stateless Chat

In the first part of the series, we successfully created a chat interface where an operator can ask questions and receive answers based on the actual content of the runbooks they have uploaded and embedded in the system. For example, they can ask in the chat, “What should I check when my server’s CPU usage exceeds 80%?” and the assistant retrieves the relevant sections from the various runbooks and assembles a coherent and concrete response.

Exploring MongoT

Thu, 28 May 2026 21:02:49 +0000

Let’s explore this fascinating and awesome Java project from MongoDB - MongoT!

You can check out the source code here:

git clone https://github.com/mongodb/mongot

MongoT is a wrapper around the amazing Java search engine: Lucene .

Lucene is a powerful search toolkit built around an inverted token index structure that enables advanced text search capabilities, including ranked results, autocomplete, synonyms, fuzzy matching, highlighting, and faceting — all with high performance regardless of dataset size. Unlike MongoDB’s native query engine, it can efficiently search across multiple indexes simultaneously by intersecting lists of ordinal document IDs in parallel, using optimization techniques like skip-lists, ordinal compression, and document frequency ordering. It also supports indexing of various field types (integers, dates, keywords, etc.) and has expanded into vector search, enabling semantic similarity search by meaning rather than exact text matching.

Implementing Soft Deletes in Java

Thu, 21 May 2026 10:09:00 +0000

What are soft deletes?

Usually, when deleting documents from a database, the entry is permanently gone and can not be recovered or accessed again.

Sometimes data needs to be made unavailable for regular access without actually being removed from a database. A common example is a user deleting their account on a platform, but retention policies require you to keep all data related to that user for a certain period of time. At the same time, no data regarding that user is accessible on the platform. A soft delete is a treatment for a piece of data that ensures it is ignored by your application while actually still being stored in the database.

Building an AI-Powered Operations Assistant with Spring AI and MongoDB Atlas — Part 1: RAG Foundation

Thu, 07 May 2026 19:22:11 +0000

This is the first article in a three-part series. Part 2 covers short-term and long-term memory; Part 3 introduces stateful workflow checkpointing with pause/resume.

The problem

It’s 2 a.m. Suddenly, an alert pops up indicating abnormal CPU usage on the payment services. The on-call engineer opens their laptop, logs into the monitoring dashboards, and begins the hunt. One by one, he searches the runbooks on Confluence, checks the Slack chats, and opens the GitHub wikis and documents shared during the design phase. By the time he finds any useful information, ten minutes have already passed.

UCanAccess: The Modern Pure-Java Bridge to Microsoft Access

Wed, 06 May 2026 08:44:00 +0000

Microsoft Access databases are everywhere. Decades of .mdb and .accdb files silently power spreadsheets, small business applications, and legacy data stores across organizations of all sizes. Yet for Java developers, connecting to these files has historically meant wrestling with native Windows libraries, ODBC bridges, and platform-specific hacks.

UCanAccess puts an end to that. It is an open-source, pure-Java JDBC driver that lets you read and write Microsoft Access databases (.mdb and .accdb) just like any other SQL database — no native drivers, no Windows dependency, no friction.

When Should You Use a Cache With MongoDB?

Tue, 05 May 2026 15:23:30 +0000

From time to time, I’ll run a design review for an application being migrated from a relational database onto MongoDB, where the customer shares an architectural diagram showing a caching layer (typically Redis) sitting between the app server and MongoDB.

I like to keep the architecture as simple as possible—after all, each layer brings its own complexity and management costs—so I’ll ask why the caching layer is there. Of course, the answer is always that it’s there to speed up data access. This reveals a misunderstanding of both the reason why caching layers were created and what MongoDB provides.

Large-Scale ETL Pipeline Architecture

Fri, 01 May 2026 19:23:04 +0000

Modern data-driven systems. ETL pipelines are no longer simply scheduled background processes that run silently overnight, one after another. They are the backbone of real-time analytics, powering operational dashboards and recommendation systems. They enable machine learning workflows.

This evolution, while creating enormous benefits, has—with the increase in data volume and the decrease in latency tolerance—called into question the traditional sequential ETL approach. A bottleneck for the speed is now required.

Building a Personalized Content Delivery System

Thu, 23 Apr 2026 15:10:45 +0000

Recommendation engines have a reputation for requiring specialized ML infrastructure: matrix factorization pipelines, training jobs, and model serving layers. That is one way to do it, but not the only way. If your data already lives in MongoDB and your application runs on Spring Boot, you can build a practical recommendation system using tools you already have. MongoDB aggregation pipelines handle the scoring math server-side, and Atlas Vector Search adds semantic matching without a separate vector database.

Distributed Cache Invalidation Patterns

Tue, 21 Apr 2026 13:52:03 +0000

Caching is one of the most powerful tools developers have at their disposal for optimizing application performance. Caching systems can significantly reduce latency and reduce the load on databases or external systems by storing frequently accessed data as close as possible to the application layer. The result? Improved responsiveness and overall system usability.

In small monolithic applications, cache management is usually very simple. A service retrieves data from a database, stores it in memory, and fulfills subsequent requests by retrieving the data directly from the cache. When the data changes, the cache key is invalidated or updated.

CQRS in Java: Separating Reads and Writes Cleanly

Thu, 16 Apr 2026 20:25:28 +0000

What you’ll learn

How the MongoDB Spring repository can be used to abstract MongoDB operations
Separating Reads and Writes in your application
How separating these can make schema design changes easier
Why you should avoid save() and saveAll() functions in Spring

The Command Query Responsibility Segregation (CQRS) pattern is a design method that segregates data access into separate services for reading and writing data. This allows a higher level of maintainability in your applications, especially if the schema or requirements change frequently. This pattern was originally developed with separate read and write sources in mind. However, implementing CQRS for a single data source is an effective way to abstract data from the application and make maintenance easier in the future. In this blog, we will use Spring Boot with MongoDB in order to create a CQRS pattern-based application.

Building a Kotlin App with Spring Boot and MongoDB Search

Thu, 09 Apr 2026 15:21:05 +0000

One of my favorite activities is traveling and exploring the world. You know that feeling of discovering a new place and thinking, “How have I not been here before?” It’s with that sensation that I’m always motivated to seek out new places to discover. Often, when searching for a place to stay, we’re not entirely sure what we’re looking for or what experiences we’d like to have. For example, we might want to rent a room in a city with a view of a castle. Finding something like that can seem difficult, right? However, there is a way to search for information accurately using MongoDB Search.

Manage HTTP Sessions with Spring Session MongoDB

Tue, 07 Apr 2026 15:19:10 +0000

Spring Session MongoDB is a library that enables Spring applications to store and manage HTTP session data in MongoDB rather than relying on container-specific session storage. In traditional deployments, session state is often tied to a single application instance, which makes scaling across multiple servers difficult. By integrating Spring Session with MongoDB, session data can be persisted beyond application restarts and shared across instances in a cluster, enabling scalable distributed applications with minimal configuration.

Java Faceted Full-Text Search API Using MongoDB Atlas Search

Thu, 02 Apr 2026 16:27:30 +0000

This is going to be a fun, practical tutorial demonstrating how to build a Java faceted full-text search API (like the ones powering sites like Amazon)!

We’ll use an interesting dataset which showcases how you can effectively pair machine learning/AI-generated data with more traditional search to produce fast, cheap, repeatable, and intuitive search engines.

TL;DR: If you (like me!) are less about words and more about code, you can jump straight in. Check it out and run it locally like this:

MongoDB Search Score Breakdown

Tue, 31 Mar 2026 15:05:08 +0000

Full-text search powers all of our digital lives — googling for this and that; asking Siri where to find a tasty, nearby dinner; shopping at Amazon; and so on. We receive relevant results, often even in spite of our typos, voice transcription mistakes, or vaguely formed queries. We have grown accustomed to expecting the best results for our searching intentions, right there, at the top.

But now it’s your turn, dear developer, to build the same satisfying user experience into your Atlas-powered application.

Modeling One-to-Many Relationships in Java with MongoDB

Thu, 26 Mar 2026 15:50:30 +0000

In a relational database, modeling a one-to-many relationship is straightforward: you create two tables and connect them with a foreign key. When you need the data together, you write a JOIN. In MongoDB, you have a choice, and that choice has a direct impact on your application’s performance, scalability, and maintainability.

Consider a common scenario: a BlogPost that has many Comment objects. In Java, this is a natural List<Comment> field on the post. But when it comes time to persist that relationship in MongoDB, you need to decide how to store it. Should the comments live inside the blog post document? Or should they sit in their own collection, connected by references?

Clean Architecture with Spring Boot and MongoDB

Tue, 24 Mar 2026 16:43:14 +0000

Most Spring Boot tutorials tightly wire everything together. Controllers call services, services call repositories, and MongoDB annotations like @Document and @Field sit right next to your business logic. It works until you need to swap the database, test logic in isolation, or reuse domain rules in a different context.

Clean Architecture enforces one rule: source code dependencies always point inward. Your business logic never imports Spring or MongoDB classes. The database becomes a pluggable detail at the outermost layer, something you can replace without rewriting core application code.

Building Reactive Data Streams with Project Reactor

Thu, 19 Mar 2026 15:13:31 +0000

Creating Non-Blocking Streaming Endpoints for High-Throughput Applications

There are problems that only occur in production. Or rather, we only notice them in production.

We have created an application that exposes clean APIs, according to all standards. You have modeled the domain elegantly and efficiently: all load tests show reassuring data. CPU usage is reasonable, to say the least. However, there is a problem: when traffic exceeds a threshold, the system slows down. Response time latency becomes inconsistent; threads stack up on top of each other. Response times increase unpredictably, and application customers begin to complain about this situation.

Language Learning Flashcard System - Part 1

Tue, 17 Mar 2026 16:39:58 +0000

My native language is Spanish. I’ve been learning (and butchering) English my whole life. But at some point, I felt confident with English and wanted to learn Japanese. Big mistake. This is a totally different beast: three writing systems (Hiragana ひらがな, Katakana カタカナ and Kanji 漢字), a completely different grammar, no relation whatsoever with European languages… I needed help. And tools to learn. And one of them is Space Repetition System based apps.

Atlas Online Archive: Efficiently Manage the Data Lifecycle

Thu, 12 Mar 2026 14:55:00 +0000

Problem statement

In the production environment, in a MongoDB Atlas database, a collection contains massive amounts of data stored, including aged and current data. However, aged data is not frequently accessed through applications, and the data piles up daily in the collection, leading to performance degradation and cost consumption. This results in needing to upgrade the cluster tier size to maintain sufficient resources according to workload, as it would be difficult to continue with the existing tier size.

Role-Based Access Control in Java Applications

Thu, 05 Mar 2026 20:31:49 +0000

We often work with Java applications where security begins and ends with authentication. The JWT token is validated, Spring Security is integrated, and an identity provider is added, thinking that this type of configuration is sufficiently secure.

The real problem is that authentication only answers one question: who are you? In real applications, we also have to answer another question, which is often more complex and more dangerous to get wrong: what are you allowed to do?

Data Enrichment in MongoDB

Tue, 03 Mar 2026 16:10:36 +0000

In a recent design review, a customer was enriching new data as it came in. As the enrichment process was fairly complex, they ran into some issues with concurrency. To solve this, they decided that data should go into a staging collection rather than the main collection that held the data. This did nothing to help with concurrency issues and actually created more work on the database side of things when enrichment was complete.

How to Build a Search Service in Java with MongoDB

Fri, 27 Feb 2026 18:28:09 +0000

We need to code our way from the search box to our search index. Performing a search and rendering the results in a presentable fashion, itself, is not a tricky endeavor: Send the user’s query to the search server, and translate the response data into some user interface technology. However, there are some important issues that need to be addressed, such as security, error handling, performance, and other concerns that deserve isolation and control.

MongoDB and the Raft Algorithm

Tue, 24 Feb 2026 15:47:27 +0000

MongoDB’s replica set architecture uses distributed consensus to ensure consistency, availability, and fault tolerance across nodes. At the core of this architecture is the Raft consensus algorithm, which breaks the complexities of distributed consensus into manageable operations: leader election, log replication, and commitment. This document explores how MongoDB integrates and optimizes Raft for its high-performance replication needs.{#1c95}

Raft Roles and MongoDB’s Replica Set

In Raft, nodes can assume one of three roles: leader , follower , or candidate . MongoDB maps these roles to its architecture seamlessly. The primary node functions as the leader, handling all client write operations and coordinating replication. The secondaries serve as followers, maintaining copies of the primary’s data. A node transitions to the candidate role during an election, triggered by leader unavailability.{#5e79}

Ports and Adapters in Java: Keeping Your Core Clean

Thu, 19 Feb 2026 16:13:59 +0000

Introduction

If we want to evolve our Java system over time, architecture is more important than the choice of framework. Very often, teams realize too late that what was supposed to be a simple persistence layer at the beginning ended up influencing and overwhelming the entire application design. MongoDB annotations end up in domain models, repository abstractions mirror collections, and business logic becomes an integral part of the infrastructure.

Hexagonal architecture, also known as Ports and Adapters, offers an architectural model that aims to avoid this problem. It involves the correct attribution of responsibilities to the various application layers. It encourages us to consider external systems (databases, message brokers, HTTP APIs, MCP servers) as details rather than pillars at the center of our design.

Optimizing the MongoDB Java Driver

Tue, 17 Feb 2026 15:33:56 +0000

Co-authored by Slav Babanin.

Donald Knuth {#https://www-cs-faculty.stanford.edu/~knuth/}, widely recognized as the ‘father of the analysis of algorithms,’ warned against premature optimization—spending effort on code that appears inefficient but is not on the critical path. He observed that programmers often focus on the wrong 97% of the codebase. Real performance gains come from identifying and optimizing the critical 3%. But, how can you identify the critical 3%? Well, that’s where the philosophy of ’never guess, always measure’ comes in.

MongoDB Sharding: What to Know Before You Shard a Collection

Thu, 12 Feb 2026 16:47:52 +0000

When we think about a system that operates at scale, we are usually talking about an application that needs to serve millions of users. This often happens when an application suddenly becomes popular and usage grows much faster than expected. As more people start using it, the system naturally begins to struggle to keep up with the load.

More users mean more requests, more data being generated, and more pressure on the database. If nothing is done, bottlenecks start to appear and the overall performance of the system degrades. There are two traditional ways to deal with this problem: vertical scaling and horizontal scaling.

Reactive Java with Project Reactor

Tue, 10 Feb 2026 14:32:03 +0000

Introduction: Why Reactive Java Still Matters

Over the past decade, the Java ecosystem has gradually abandoned the idea that increasing the number of threads is the scalable solution to growing load. Cloud-native implementations, containerized workloads, and high-I/O applications have highlighted the inefficiencies of the traditional synchronous thread-per-request model.

Reactive programming is not a panacea, a miracle solution. Certainly, it does not make applications “faster” by default. What it offers, when applied correctly, is the ability to predict behavior under load, better resource utilization, and explicit control over data flow. For systems that handle high concurrency, streaming data, or variable traffic patterns, these characteristics are hugely relevant.

MongoDB 8.0 Migration Guide: What You Need to Know Before Upgrading

Thu, 05 Feb 2026 14:56:13 +0000

Have you ever wondered why updates and upgrades are so essential for any system? Well, it’s no secret: They ensure that systems remain relevant and efficient. With MongoDB, it’s no different. Whenever we think about updating, we seek efficiency, security, performance, and other benefits that come with updated systems. However, every update introduces changes that need to be carefully managed. In this article, we will cover some of the new features of MongoDB version 8.0 and highlight the key considerations you should take into account before migrating to this new version.

GraphQL for Java Developers: Building a Flexible Data Layer

Tue, 03 Feb 2026 15:43:07 +0000

For many years, REST has been the standard architectural style for creating APIs in the Java ecosystem. Frameworks such as Spring MVC and, more recently, Spring WebFlux make it easy to expose HTTP endpoints with the REST paradigm, supported by well-structured service layers. In many cases, this model works well and serves as the basis for numerous enterprise solutions.

However, as applications grow and frontend requirements become more dynamic, REST-based APIs are beginning to show their limitations. Multiple endpoints that return rigid DTOs often lead to over-fetching, under-fetching, and a proliferation of specialised endpoints created to meet slightly different customer needs. Over time, APIs become more difficult to evolve without introducing radical changes.

Abstracting Data Access in Java With the DAO Pattern

Thu, 29 Jan 2026 16:37:06 +0000

The Data Access Object (DAO) pattern is a structural pattern that isolates your application’s business logic from persistence operations. By using an abstract API, the DAO pattern hides all the complexity of performing CRUD operations against your database—whether that’s MongoDB, a relational database, or any other storage mechanism.

This separation is crucial: Your business logic shouldn’t care whether you’re using MongoDB’s flexible document model or a rigid SQL schema. The DAO pattern ensures both layers can evolve independently.

Introduction to Behavior Driving Development with Java and MongoDB

Tue, 27 Jan 2026 16:35:53 +0000

When we face software development, the biggest mistake is about delivering what the client wants. It sounds like a cliché, but after decades, we are still facing this problem. One good way to solve it is to start the test focusing on what the business needs.

Behavior-driven development (BDD) is a software development methodology where the focus is on behavior and the domain terminology or ubiquitous language. It utilizes a shared, natural language to define and test software behaviors from the user’s perspective. BDD builds upon test-driven development (TDD) by focusing on scenarios that are relevant to the business. These scenarios are written as plain-language specifications that can be automated as tests, simultaneously serving as living documentation.

MongoDB and WiredTiger: A Journey Through the Storage Engine

Thu, 18 Dec 2025 20:26:28 +0000

Databases are the backbone of modern applications, and MongoDB stands out with its flexibility and scalability. Central to its functionality is the WiredTiger storage engine. WiredTiger, as MongoDB’s default engine, seamlessly merges document-level concurrency for high throughput, advanced compression techniques for optimized storage, and an in-memory architecture for rapid data access.{#e668}

With the addition of write-ahead logging for robust durability and the sophistication of MultiVersion Concurrency Control for snapshot-like data views, WiredTiger harmoniously orchestrates MongoDB’s data management.{#6fac}

Inside the Engine: Performance Relay of MongoDB 8.0

Tue, 16 Dec 2025 20:16:42 +0000

In environments where microseconds dictate competitive advantage, MongoDB 8.0 delivers a meticulously tuned execution pipeline that transforms raw network packets into sub-millisecond query responses at global scale.{#e2e4}

This reference traces a single trade query through every internal boundary network ingress, scheduling, security, parsing, planning, execution, storage‐engine internals, indexing, replication, sharding, change streams, time‐series buckets, backup, and monitoring illustrating how MongoDB 8.0’s per-CPU allocators, active-work profiling, SIMD-vectorized execution, adaptive bucketization, compact resume tokens, and refined journaling coalesce into a seamless, predictable performance engine.{#e2e4}

BoxLang Native Couchbase Module

Tue, 16 Dec 2025 08:37:02 +0000

Ortus Solutions is thrilled to announce the official release of bx-couchbase v1.0 , a groundbreaking module that brings native enterprise-grade Couchbase capabilities to the BoxLang language and ecosystem. Designed for modern distributed applications, bx-couchbase unifies high-performance caching, resilient distributed locking, and advanced AI vector memory—empowering developers to build scalable, intelligent, fault-tolerant systems with unprecedented ease.

Couchbase + BoxLang: A High-Performance Combination

Couchbase is more than a database—it’s a distributed NoSQL platform engineered for speed, flexibility, and global scale. With sub-millisecond key-value operations, built-in vector search, and multi-data-center support, it is the ideal foundation for next-generation AI-powered applications.

Building Java Microservices with the Repository Pattern

Thu, 11 Dec 2025 23:19:10 +0000

What you’ll learn

How the MongoDB Spring repository can be used to abstract MongoDB operations
Ensuring data access is separate from core application logic
Why you should avoid save() and saveAll() functions in Spring
Why schema and index design still matters in this case

The repository pattern is a design method that allows for abstraction between business logic and the data of an application. This allows for retrieving and saving/updating objects without exposing the technical details of how that data is stored in the main application. In this blog, we will use Spring Boot with MongoDB in order to create a repository pattern-based application.

Optimizing MongoDB Queries in Java Applications

Tue, 09 Dec 2025 17:45:09 +0000

Modern Java applications often struggle with performance bottlenecks that have little to do with the JVM itself. In most cases, the culprit lies deeper in how the application interacts with its database. Slow queries, missing indexes, or inefficient access patterns can quietly degrade user experience, increase latency, and inflate infrastructure costs. MongoDB, known for its flexibility and document-oriented design, can deliver remarkable performance when used correctly. However, that performance can quickly diminish when queries and indexes are not aligned with real-world access patterns.

Building Systems That Know Why They Exist ~ When Data, Logic, and Intent Finally Align

Thu, 04 Dec 2025 19:06:35 +0000

Every software system begins with intent. A human decision. A statement of what should exist, how it should behave, and why it matters. But somewhere along the way, that intent dies. It’s decomposed into documentation, user stories, and scattered logic. It becomes a shadow of itself fragmented across layers of code, processes, and people who no longer remember what it was meant to achieve.{#7cd1}

We’ve automated infrastructure. We’ve scripted configurations. We’ve even made AI write code for us. But we’ve never automated understanding . Our systems execute brilliantly yet comprehend nothing. They operate without memory of purpose. They run, but they do not know why.{#657d}

Atlas Searching with the Java Driver

Tue, 02 Dec 2025 17:28:43 +0000

Dear fellow developer, welcome!

Atlas Search is a full-text search engine embedded in MongoDB Atlas that gives you a seamless, scalable experience for building relevance-based app features. Built on Apache Lucene, Atlas Search eliminates the need to run a separate search system alongside your database. The gateway to Atlas Search is the $search aggregation pipeline stage.

The $search stage, as one of the newest members of the MongoDB aggregation pipeline family, has gotten native, convenient support added to various language drivers. Driver support helps developers build concise and readable code. This article delves into using the Atlas Search support built into the MongoDB Java driver, where we’ll see how to use the driver, how to handle `$search` features that don’t yet have native driver convenience methods or have been released after the driver was released, and a glimpse into Atlas Search relevancy scoring. Let’s get started!

Navigating the Nuances of GraphRAG vs. RAG

Thu, 20 Nov 2025 21:28:52 +0000

While large language models (LLMs) hold immense promise for building AI applications and agentic systems, ensuring they generate reliable and trustworthy outputs remains a persistent challenge. Effective data management—particularly how data is stored, retrieved, and accessed—is crucial to overcoming this issue. Retrieval-augmented generation (RAG) has emerged as a widely adopted strategy, grounding LLMs in external knowledge beyond their original training data.

The standard, or baseline, implementation of RAG typically relies on a vector-based approach. While effective for retrieving contextually relevant documents and references, vector-based RAG faces limitations in other situations, particularly when applications require robust reasoning capabilities and the ability to understand complex relationships between diverse concepts spread across large knowledge bases. This can lead to outputs that disappoint or even mislead end-users.

Service Layer Pattern in Java With Spring Boot

Tue, 18 Nov 2025 14:40:26 +0000

In modern software design, it is important to develop code that is clean and maintainable. One way developers do this is using the Service Layer pattern.

What you’ll learn

In this article, you’ll learn:

What the Service Layer pattern is and why it matters.
How it fits with the MVC architecture.
How to implement it in a real Spring Boot application.
How to add MongoDB with minimal code.
Best practices and common mistakes to avoid.

What is the Service Layer pattern?

The Service Layer pattern is an architectural pattern that defines an application’s boundary with a layer of services that establishes a set of available operations and coordinates the application’s response in each operation.

Beyond Keywords: Hybrid Search With Atlas and Vector Search (Part 3)

Thu, 13 Nov 2025 13:56:23 +0000

Bringing together semantic vectors and exact keyword matching with $rankFusion

If you’ve been following along this series, you already know we started by giving our movie search app the ability to understand meaning—not just keywords—using semantic search, as discussed in Part 1: Implementing Semantic Search in Java With Spring Data . Then, we made it even smarter by adding filters and optimizing performance with embedding strategies in Part 2: Optimizing Vector Search With Filters and Caching .

The Cost of Not Knowing MongoDB - Part 3: (appV6R0 to appV6R4)

Tue, 11 Nov 2025 19:31:10 +0000

Welcome to the third and final part of the series “The Cost of Not Knowing MongoDB.” Building upon the foundational optimizations explored in Part 1 and Part 2 , this article delves into advanced MongoDB design patterns that can dramatically transform application performance.

In Part 1 , we improved application performance by concatenating fields, changing data types, and shortening field names. In Part 2 , we implemented the Bucket Pattern and Computed Pattern and optimized the aggregation pipeline to achieve even better performance.

Modeling Relationships With Hibernate ORM and MongoDB

Tue, 04 Nov 2025 17:08:49 +0000

In theprevious article —Getting Started With Hibernate ORM and MongoDB—we learned how to configure Hibernate to work with MongoDB, create an entity, and perform basic CRUD operations using the familiar Hibernate API.

If you haven’t read that first part yet, I recommend starting there before continuing. It covers the project setup, dependencies, and the fundamentals that we’ll build upon here.

In this second part, we’ll extend our application to model relationships between entities—introducing a Review entity and linking it to our existing Book class. This will allow us to explore more advanced capabilities of the MongoDB Hibernate integration, including:

Getting Started With Hibernate ORM and MongoDB

Tue, 04 Nov 2025 17:06:26 +0000

For years, Hibernate ORM has been one of the most popular frameworks in the Java ecosystem. It was built to simplify data persistence by letting developers work with Java objects instead of SQL statements, a technique known as object-relational mapping (ORM).

Traditionally, Hibernate ORM has been tightly associated with relational databases like PostgreSQL, MySQL, and Oracle. It manages connections, transactions, and entity state behind the scenes, and even provides Hibernate Query Language (HQL) so you can query your data using Java entity names rather than table names.

The Pitfall of Increasing Read Capacity by Reading From Secondary Nodes in a MongoDB Replica Set

Thu, 30 Oct 2025 15:23:31 +0000

The scenario

Imagine we are responsible for managing the MongoDB cluster that supports our country’s national financial payment system, similar toPix in Brazil. Our application was designed to be read-heavy, with one write operation for every 20 read operations.

WithBlack Friday approaching, a critical period for our national financial payment system, we have been entrusted with the crucial task of creating a scaling plan for our cluster to handle the increased demand during this shopping spree. Given that our system is read-heavy, we are exploring ways to enhance the read performance and capacity of our cluster.

How MongoDB Decides What to Forget?

Tue, 28 Oct 2025 14:00:39 +0000

Inside MongoDB’s storage engine, WiredTiger , nothing happens by accident.{#4cf7}

Every page in memory exists under policy — governed, measured, and continuously evaluated against the limits of RAM, I/O bandwidth, and checkpoint cadence. Eviction is not cleanup. It’s runtime arbitration between volatility and durability.{#ba51}

When the process starts, WiredTiger allocates a fixed memory region known as the cache arena typically 50% of physical RAM. Within that space live B-tree pages : internal nodes, leaf nodes, and history-store entries. Each page carries operational metadata: dirty, clean, hazard-protected, in-use, last_access_time, and generation . This metadata feeds into a per-page score, which informs the eviction subsystem’s next decision.{#61cb}

Beyond Keywords: Optimizing Vector Search with Filters and Caching (Part 2)

Thu, 23 Oct 2025 15:44:38 +0000

Enhancing precision with pre-filters and reducing costs with embedding caching

Welcome back! If you landed here without reading Part 1: Beyond Keywords: Implementing Semantic Search in Java With Spring Data , I recommend going back and checking it first so the steps in this article make more sense in sequence.

This is the second part of a three-part series where we’re building a movie search application. So far, our app supports semantic search using vector queries with Spring Data and Voyage AI. In this article, we’ll take things further:

The Cost of Not Knowing MongoDB - Part 2

Tue, 21 Oct 2025 14:26:51 +0000

This is the second part of the series “The Cost of Not Knowing MongoDB,” where we go through many ways we can model our MongoDB schemas for the same application and have different performances. In the first part of the series, we concatenated fields, changed data types, and short-handed field names to improve the application performance. In this second part, as discussed in the issues and improvement of appV4 , the performance gains will be achieved by analyzing the application behavior and how it stores and reads its data, leading us to the use of the Bucket Pattern and the Computed Pattern .

Beyond Keywords: Implementing Semantic Search in Java With Spring Data (Part 1)

Thu, 16 Oct 2025 13:39:59 +0000

Building a semantic movie search app with embeddings and vector queries

Have you ever tried to search for something such as a product, a song, or a movie but couldn’t quite remember its exact name? Maybe you recall only a clue—a desert pyramid, a short melody, or “that ship that hit an iceberg.” Keyword search struggles with that. Vector search doesn’t: It lets you search by meaning.

It works by turning text into embeddings, vectors (arrays of numbers) that capture semantic similarity, so results are ranked by what they mean, not just what they say.

The Cost of Not Knowing MongoDB (Part 1)

Tue, 14 Oct 2025 13:43:42 +0000

The primary focus of this series is to show how much performance you can gain, and as a consequence, the cost you can save when using MongoDB properly, following the best practices, studying your application needs, and using it to model your data.

To show these possible gains, a dummy application will be presented, and many possible implementations of it using MongoDB will be developed and load-tested. There will be implementations for all levels of MongoDB knowledge: beginner, intermediate, senior, and mind-blowing (🤯) .

Keywords Meet Vectors: Hybrid Search on MongoDB

Thu, 09 Oct 2025 14:49:49 +0000

In the previous issues, I explained how to run a local MongoDB Atlas cluster using Atlas CLI , what vector search is, and how to use it . Now, let’s take a closer look at hybrid search.

Reading time: 4–5 min

What is hybrid search?

Hybrid search in MongoDB brings together two complementary search techniques:

Full text search (BM25 via Atlas Search )—optimized for exact keyword matches, powered by Lucene inside mongot. Perfect when users expect documents that literally contain their query terms.

Vector search (kNN via Atlas Vector Search )—optimized for semantic similarity. It uses dense embeddings from ML models to find conceptually related content, even when no keywords match.

On their own, each method has advantages and limitations. Text search misses context (“non-linear crime story” won’t return Memento). Pure semantic search may return results that are semantically aligned but sometimes not practically useful. Hybrid search combines the strengths of both, ensuring results are contextually relevant and precise.

Working with Geo Location Data in MongoDB

Tue, 07 Oct 2025 14:01:53 +0000

MongoDB makes it really easy to work with location data (sometimes called Geo Data) by simplifying how to store this type of data and streamlining how you query for it so you can easily create “find nearby” queries, or plot your location data with ease!

Let’s start with the basics: modeling your data, indexing it properly, running geo queries, and then displaying results on a map.

Model Your Data with GeoJSON

Agents Meet Databases: The Future of Agentic Architectures

Thu, 02 Oct 2025 15:01:49 +0000

With 2025 hailed as “the year of agents” by NVIDIA CEO Jensen Huang and OpenAI CPO Kevin Weil , AI agents are increasingly of interest to organizations across industries. These autonomous systems will often need to interact with databases, where much of the world’s valuable data resides. According to IDC’s Data Age 2025 report , enterprises will manage nearly 60% of the world’s data by 2025, most of it organized in databases. As a result, databases will be central to agentic architectures, and the success of agent deployments will depend on how well they connect and interact with them.

Introduction to Data-Driven Testing with Java and MongoDB

Thu, 25 Sep 2025 13:53:57 +0000

As applications expand, the complexity of the rules they enforce also increases. In many systems, these rules are embedded within the data, primarily in database queries that filter, join, or compute based on real-world conditions. However, the tests for these queries are often shallow, repetitive, or, worse yet, completely absent. When there is an error in the database logic, the application may still compile successfully, but the business can suffer significant consequences.

Power your AI application with Vector Search

Tue, 23 Sep 2025 16:23:37 +0000

Most major database vendors, like MongoDB , are adding vector search capabilities to their products. It’s becoming a standard feature as demand for AI-powered applications grows.

🕒 Reading time: 3-4 min

🧠 What is vector search needed for?

MongoDB Vector Search enables semantic queries. For example, finding support tickets with similar meanings even if they use different words. It also powers hybrid search by combining exact keyword matches like “error 500” with semantically similar phrases like “server failure”. Another use case is personalization, such as recommending articles similar to those a user has read.

From Zero to Vector Hero - Locally! (Vector Search)

Thu, 18 Sep 2025 15:25:23 +0000

In the previous issue, I explained how to run a local MongoDB Atlas cluster using Atlas CLI - no cloud account required. If you missed it, read it here 👉 Run an Atlas cluster locally in minutes. Now let’s see how to use Vector Search in that local environment.

🕒 Reading time: 3-4 min

🧠 Explaining the Embedding Workflow

After launching your local MongoDB Atlas cluster and running the show dbs command in mongosh, you’ll see only the default system databases: admin, config, and local. These are used internally by MongoDB and contain no user data or vector embeddings at this point.

Best Practices for Deploying MongoDB in Kubernetes

Tue, 16 Sep 2025 19:01:34 +0000

Kubernetes , also known as K8s, is an open-source system that simplifies the deployment, scaling, and management of containerized applications. You define your application’s desired state, such as the number of instances (pods ) or how they communicate, and Kubernetes works continuously to ensure that state is met. It excels at running stateless workloads, where pods can be replaced at any time without impacting application state or user data. Think of web servers or REST APIs: If a pod crashes, Kubernetes simply spins up a replacement and everything continues as expected.

MongoDB Schemas in Java

Thu, 11 Sep 2025 13:15:02 +0000

If (like me!) you prefer to read code rather than a long and boring article, jump straight to the example! TLDR; here’s the code .

You can clone and run the tests like this:

git clone https://github.com/luketn/mongodb-schemas-in-java 
cd mongodb-schemas-in-java 
mvn test

And if you want to play with the example API visually (like I do!) and you have a local Mongo or an Atlas Cluster with the Weather Sample data, run:

Building REST APIs in Java with Spring Boot

Tue, 09 Sep 2025 14:51:10 +0000

REST has become the default choice for building web services, and for good reason. It’s straightforward to implement, easy for clients to consume, and built directly on top of the same principles of the web itself.

HTTP already gives us well-defined methods (GET, POST, PUT, DELETE), built-in caching, redirect support, secure transport via TLS, and widespread tooling support across platforms. REST doesn’t reinvent the web—it uses it.

REST is not a protocol or a rigid standard. It’s a lightweight architectural approach that encourages scalable, evolvable, and interoperable services. Its creator, Roy Fielding, helped define many of the specs that underpin the web today.

MongoDB Aggregations: Organizing Recipes by Meal Type with $group

Thu, 04 Sep 2025 05:31:22 +0000

In this series, we’re exploring different MongoDB aggregation operators by applying them to a recipe collection. I hope you’ll follow along with each post!

MongoDB Aggregations Series

MongoDB Aggregations: Finding Cooking Times with $min and $max
MongoDB Aggregations: Organizing Recipes by Meal Type with $group

As part of this series, we are imagining you’re building a recipe website. For this post, we are adding some new features to our site, including allowing users to browse recipes by meal type—breakfast, lunch, dinner, etc.

Copy of Queryable Encryption with Spring Data MongoDB: How to Query Encrypted Fields

Tue, 02 Sep 2025 19:00:24 +0000

Information is one of the most valuable assets in computing and keeping it protected is even more critical. When we talk about data protection, it’s not just about preventing breaches or leaks; it’s also about complying with privacy regulations and protecting user data.

MongoDB provides strong encryption capabilities, including in transit, at rest, and in use. The Queryable Encryption feature falls into the in use category. It allows data to be encrypted on the client side, so that even with access to the database and its credentials, no one can read the protected fields without the proper encryption key. At the same time, it supports querying over encrypted fields, making it possible to filter, match, or retrieve data without compromising confidentiality.

What happens when a Primary fails in a MongoDB replica set?

Tue, 26 Aug 2025 15:14:03 +0000

In distributed systems, high availability is not a luxury—it’s a necessity. And one of the very important parts of that is automatic failovers. What are automatic failovers and how do they work? Let’s see today!

These concepts are actually similar in every database, but as MongoDB is natively built for replica-sets, we will talk about that here, whereas other DB systems need 3rd-party or managed systems to implement the same.

Managing MongoDB Database Changes Using Liquibase Pro

Wed, 20 Aug 2025 14:21:51 +0000

This article explores how Liquibase and MongoDB can be integrated with Git actions to implement source control for database changes.

What is Liquibase?

Liquibase is an open-source tool designed to manage version control for database schema changes. It allows developers to simplify and automate the process of applying database updates across various database platforms.

By using database-independent formats like XML, YAML, or JSON, Liquibase enables teams to define changes in a consistent and structured way. In a Java project, these updates are organized into files containing changesets, which outline the specific instructions needed to modify or refactor the database schema.

Domain-Driven Design in Java: A Practical Guide

Fri, 15 Aug 2025 09:30:31 +0000

While waiting for your flight in an airport, have you ever wondered how much behind-the-scenes planning it takes to keep an airport running smoothly? Every day, thousands of flights take off and land, passengers rush through terminals, and staff work tirelessly to make sure everything unfolds without a hitch. What looks like chaos on the surface is actually the result of a highly coordinated system—a kind of “brain” that’s always working in the background to manage logistics, adapt to surprises, and keep the airport humming along.

Enforcing Governance in MongoDB Atlas with Resource Policies

Wed, 13 Aug 2025 12:52:05 +0000

The growing adoption of MongoDB Atlas as a managed database platform increasingly demands mechanisms that align security, compliance, and governance. In April 2025, MongoDB introduced a critical feature for this purpose: Resource Policies .

This article presents the concept and capabilities of Resource Policies, highlights their importance in enterprise environments, and demonstrates how to use them effectively via Terraform — ensuring that essential configurations rely not only on best practices but also on enforceable and auditable constraints.

Why Mirroring Production in Dev Helps You Avoid Costly Mistakes

Mon, 28 Jul 2025 13:31:10 +0000

Many developers start building their applications with MongoDB using a free M0 cluster or a local environment. While this is common and convenient, it can lead to issues that could easily be avoided by using a more robust setup, such as a development cluster that closely mirrors production. Problems like inefficient queries, missing indexes, or even costly mistakes often go unnoticed in limited environments like M0 or local setups.

In this article, we’ll explore why your development environment should closely reflect your production environment, and how using an M10+ cluster with tools like Query Profiler and Performance Advisor can help you catch performance issues early and build with confidence.

MongoDB ACID Transactions With Java

Wed, 23 Jul 2025 14:39:45 +0000

A database is constantly under pressure. Real-time applications, exponential growth of data, and multiple operations may hit the system at once. Without coordination, this can lead to race conditions, conflicts, and ultimately, the dreaded inconsistent data! To address this, MongoDB supports transactions .

Transactions let us group multiple operations together so they either all succeed or none take effect, a concept known as atomicity . MongoDB’s document model already provides atomicity at the single-document level, which covers a lot of use cases. But when you need atomicity across multiple documents or collections, transactions are the answer.

How to Make a RAG Application With LangChain4j

Thu, 17 Jul 2025 20:42:12 +0000

Retrieval-augmented generation, or RAG, introduces some serious capabilities to your large language models (LLMs). These applications can answer questions about your specific corpus of knowledge, while leveraging all the nuance and sophistication of a traditional LLM.

This tutorial will take you through the ins and outs of creating a Q&A chatbot using RAG. The application will:

Retrieve data from a MongoDB Atlas database.
Embed and store documents as vector embeddings.
Use LangChain4J to query the database and augment LLM prompts with the retrieved data.
Enable secure, scalable, and efficient AI-powered applications.

If you want to see the completed application, it is available in the GitHub repository .

The Great Data Reimagination: From Static to Agile in the AI Era

Tue, 15 Jul 2025 13:44:06 +0000

We’re in the middle of a fundamental change in how enterprise software works. In the next decade, your database will become your AI.

Just five years ago, choosing the right kind of database to support their applications presented complexity for many developers: relational or NoSQL? Structured or unstructured? Flexible or predictable? They didn’t know exactly what their data would look like in six months, but what they did know was that it was certain to change. This led many to make a rebel’s choice to reject the rigid structure of SQL databases for something more fluid and adaptable.

Building a Spring Boot CRUD Application Using MongoDB’s Relational Migrator

Tue, 08 Jul 2025 19:05:43 +0000

Imagine this: You’re working with a relational database that’s served you well for years, but now your applications demand flexibility, scalability, and faster development cycles. Transitioning from structured tables to MongoDB’s dynamic, schema-less JSON documents might be daunting. That’s where MongoDB’s Relational Migrator becomes your secret weapon.

The Relational Migrator tool built by MongoDB no longer only supports migrating data from relational databases to MongoDB. Rather, it supports features like code generation and also a query converter that converts a relational database, such as a Postgres query, or queries from views and stored procedures, into a MongoDB query.

Data Modeling for Java Developers: Structuring With PostgreSQL and MongoDB

Thu, 03 Jul 2025 21:02:16 +0000

Application and system designs have always been considered the most essential step in application development. All the later steps and technologies to be used depend on how the system has been designed. If you are a Java developer, choosing the right approach can mean distinguishing between a rigid, complex schema and a nimble, scalable solution. If you are a Java developer who works with PostgreSQL or other relational databases, you understand the pain of representing the many-to-many relationships between the tables.

Java Virtual Threads in Action: Optimizing MongoDB Operation

Tue, 01 Jul 2025 08:12:24 +0000

Virtual threads have become one of the most popular resources in Java and are trending inside the language. Indeed, this resource introduced a cheap way to create threads inside the JVM. In this tutorial, we will explain how to use it with MongoDB.

You can find all the code presented in this tutorial in the GitHub repository :

git clone [email protected]:soujava/mongodb-virtual-threads.git

Prerequisites

For this tutorial, you’ll need:

How to Identify the Underlying Causes of Connection Timeout Errors for MongoDB With Java

Thu, 26 Jun 2025 12:13:44 +0000

Java developers and MongoDB are like Aladdin and the Genie from Arabian Nights. Developers rub the lamp with their wildest NoSQL wishes, and MongoDB swoops in, granting Spring Boot microservices and REST APIs the magic they need to soar. But every so often, a Jafar-like menace swoops in, forcing our Aladdin (Java devs) to wrestle with sleepless nights. One such villainous foe is the connection timeout, locking APIs in a cave of wonders with no escape, leaving developers yearning for a magic carpet fix.

Clean and Modular Java: A Hexagonal Architecture Approach

Tue, 24 Jun 2025 08:03:50 +0000

One of the discussions that always leaves me with both doubts and excitement is the one about system architecture. Ever since I started diving deeper into programming, I’ve encountered questions like how to separate packages and modules: is it really worth creating so many divisions? I must admit, it’s often complicated to understand or make decisions without being fully sure that I’m doing the right thing.

Right from the start, I want to make it clear that this content will definitely spark a lot of discussion around best practices and whether what I’m doing is correct — but, above all, I guarantee it will be fun!

Multi-cloud Strategies With MongoDB Atlas

Thu, 19 Jun 2025 11:49:29 +0000

In the technological world, the cloud has become more prevalent. It brings many benefits, including flexibility, scalability, faster innovation, and collaboration. Plus, when it comes to data storage and access with databases, it allows data to be located closer to the user for lower latency and thus, faster performance.

Most people have heard of the big cloud providers, such as Azure from Microsoft, Google Cloud, and Amazon Web Services (AWS). But when it comes to a business selecting a cloud provider, how do you choose?

Your Complete Guide to Diagnose Slow Queries in MongoDB

Tue, 17 Jun 2025 04:06:04 +0000

MongoDB is built to be fast. The real win comes from knowing how to keep it that way, even as your app grows and your data gets more complex.

This tutorial is for those developers and engineers who want to understand how MongoDB behaves when queries or writes slow down. We’re not here to hand you indexing tips or premature advice about schema design. We’re here to walk through the actual tools MongoDB gives you to observe, measure, and reason about query performance, before you start optimizing anything.

Performance Best Practise No. 1: Optimize Database Operations

Wed, 11 Jun 2025 13:24:32 +0000

Database operations are a very critical part of most applications in regards of performance. There are multiple reasons why database operations can significantly contribute to lower performance:

The database often runs on a remote server, slowing down communication with the database and the data transfer
Establishing individual connections to a database can take a significant portion of time compared to running the whole database query
Database queries can run for a long time
Network communication is unstable and may required restarting queries in case of network failures

You can address the above issues and boost Jakarta EE database performance by leveraging the following best practices.

Intro to RAG: Foundations of Retrieval Augmented Generation, part 2

Tue, 10 Jun 2025 21:54:49 +0000

In the last post , we discussed the basics of Retrieval Augmented Generation (RAG) and how it enhances the capabilities of Large Language Models (LLMs) by integrating them with external knowledge sources. We also introduced the concept of vector embeddings and their role in semantic search.

In this post, we’ll dive deeper into the different layers of RAG, including vector RAG, graph RAG, and agents. We’ll explore how these layers can be combined to create more powerful and effective AI systems.

MongoDB Aggregation Framework: A Beginner’s Guide

Thu, 05 Jun 2025 06:20:02 +0000

Finding exactly the data we need isn’t always a simple task.

You’ve probably faced situations where you needed to filter information, group it, and even perform calculations to produce a final result.

And often, delivering this processed data to the client is essential for the application’s success. MongoDB offers two main ways to fetch data:

Testing MongoDB Atlas Search Java Apps Using TestContainers

Thu, 29 May 2025 18:07:27 +0000

This will be an exploration of testing MongoDB Atlas Search solutions written in Java using TestContainers and JUnit5. We’ll start simple and build up to more advanced uses which load seed data and provide an environment for consistent and easy to maintain tests.

TLDR; if (like me!) you want to get straight to the code rather than reading lots of tedious words:

git clone https://github.com/luketn/mongodb-atlas-local-testcontainers.git
cd mongodb-atlas-local-testcontainers
mvn test

What is MongoDB Atlas Search, anyway?

MongoDB Atlas Search is an extension to the built-in indexing capabilities that are part of MongoDB itself, using the awesome open source indexing and query library Lucene . MongoDB has built a wrapper around Lucene called mongot .

Intro to RAG: Foundations of Retrieval Augmented Generation, part 1

Tue, 27 May 2025 14:42:43 +0000

Retrieval Augmented Generation (RAG) may sound complex, but it accurately represents the process of the system. RAG is a method that enhances the capabilities of Large Language Models (LLMs) by integrating them with external knowledge sources.

Each term represents a piece of the puzzle:

Retrieval - data retrieved from some external source outside the LLM (most often a database, but can include files, webpages, etc)
Augmented - “augmenting” (or adding to) an LLM’s training data. This could include recent or private information that it did not have access to during its training period. Most often, this is done by adding the data to the prompt (or input) to the LLM.
Generation - this is where LLMs are exceptional. They generate a response (text, image, video, etc) that is similar to the data being provided in the input. This is generated from probabilities, so cannot guarantee 100% consistency.

Instead of relying solely on the model’s internal training data, RAG retrieves relevant information from databases or document collections to ground its responses in factual and up-to-date content. This approach not only improves the accuracy and reliability of the generated outputs but also allows the system to adapt to specific contexts or domains, making it a powerful tool for many personal and professional applications.

Understanding BSON: A Beginner’s Guide to MongoDB’s Data Format

Thu, 22 May 2025 08:05:06 +0000

When working with MongoDB, it’s easy to think you’re dealing with JSON. After all, the queries, documents, and API responses all look like JSON. But MongoDB is not storing JSON. It’s storing BSON—a binary format designed for efficient storage and fast traversal.

BSON (Binary JSON) is more than just a binary version of JSON. It introduces additional data types like ObjectId, Decimal128, and Timestamp, allowing MongoDB to handle more complex data structures and ensure data integrity. While we might rarely interact with raw BSON directly, understanding how MongoDB stores and processes BSON documents can help us write more efficient queries, handle data conversions properly, and debug unexpected behavior.

Building a Real-Time AI Fraud Detection System with Spring Kafka and MongoDB

Wed, 14 May 2025 14:30:08 +0000

In this tutorial, we’ll build a real-time fraud detection system using MongoDB Atlas Vector Search, Apache Kafka, and AI-generated embeddings. We’ll demonstrate how MongoDB Atlas Vector Search can be used to detect anomalies in a stream of financial transactions by analyzing a user’s transaction history and identifying suspicious behavior based on LLM-generated embeddings.

Our solution will monitor MongoDB Change Streams using the Java synchronous driver, triggering vector searches on each new transaction to detect potential fraud. While this approach works well for our demo, and for many use cases, we’ll also discuss its limitations. Throughout the tutorial, I’ll cover alternative strategies to optimize performance, whether you need higher transaction throughput, faster fraud detection, or a more scalable architecture.

GenAI blood, sweat, and tears: Loading data to Pinecone

Wed, 07 May 2025 22:10:37 +0000

As someone who is pretty familiar with relational and graph databases, I wanted to dig a little deeper into vector databases and understand the strengths and quirks they bring to the database table. I put together a conference abstract on vector RAG versus GraphRAG which got picked up, so I went to work building a demo and learning all I could.

I pivoted a few times along the way, but ended up with a Spring AI application that connects to both Pinecone (vector database) and Neo4j (graph database). There were a few surprises throughout, which I’ll be sure to mention. Let’s get started!

Semantic Search with Spring Boot & Redis

Mon, 05 May 2025 15:35:55 +0000

TL;DR: You’re building a semantic search app using Spring Boot and Redis. Instead of matching exact words, semantic search finds meaning using Vector Similarity Search (VSS). It works by turning movie synopses into vectors with embedding models, storing them in Redis (as a vector database), and finding the closest matches to user queries.

A traditional searching system works by matching the words a user types with the words stored in a database or document collection. It usually looks for exact or partial matches without understanding the meaning behind the words.

Breaktime Tech Talks (Ep39): Why embedding models should match + Advice for starting a blog

Wed, 16 Apr 2025 05:35:04 +0000

This is episode 39 of the Breaktime Tech Talks podcast .

I’m at a conference this week presenting on a variety of different topics, but I stopped for just a few minutes to document some recent learnings while I was preparing the code projects for this week’s sessions.

I’ll chat about a hands-on experience that helped me understand why embedding models for vectors and search should actually match. Then I came across an article with advice for creating a blog that outlines a lot of the practices I myself follow when I write tech blogs. I hope you enjoy it as well.

Breaktime Tech Talks (Ep37): Vector Database Frustration + Microsoft LazyGraphRAG

Wed, 19 Mar 2025 12:54:36 +0000

This is episode 37 of the Breaktime Tech Talks .

You are listening to the Breaktime Tech Talks podcast, a bite-sized tech podcast for busy developers, where we’ll briefly cover technical topics, new snippets, and more in short time blocks. I’m your host, Jennifer Reif, an avid developer and problem solver with special interest in data, learning, and all things technology.

Book Review: DuckDB in Action

Wed, 05 Mar 2025 13:42:59 +0000

Disclaimer: this post includes affiliate links; I may receive compensation if you purchase the book from the different links provided in this post.

This review is about DuckDB in Action by JoMark Needham, Michael Hunger, and Michael Simons from Manning.

The book was sent to me by Michael Simons . He asked for my feedback: I changed my reading schedule, took a few months, and here it is.

Facts

10 chapters
288 pages
$33.59 (eBook)

Note that MotherDuck, the company behind DuckDB, offers a free PDF copy .

Sliding Window Counter Rate Limiter (Redis & Java)

Tue, 25 Feb 2025 21:21:11 +0000

This content is also available on YouTube. Check it out!

The Sliding Window Counter offers a more efficient way to handle rate limiting compared to the Sliding Window Log. While the Sliding Window Log keeps an exact log of timestamps for each request, allowing precise tracking over a rolling time period, this precision comes at the cost of higher memory usage and computational overhead as the number of requests increases.

Spring Data Neo4j: How to update an entity

Wed, 05 Feb 2025 18:58:12 +0000

After working on a new online Spring Data Neo4j course , I learned a couple more things about updating an entity.

The course required a different set of scenarios than outlined in my previous SDN update blog post , so I wanted to cover those scenarios, as well.

Spring save() method

First up is the out-of-the-box save() method that is provided by Spring as a default. This method takes an input of the entity object you want to save to the database.

Token Bucket Rate Limiter (Redis & Java) - Raphael De Lio

Tue, 28 Jan 2025 07:07:08 +0000

This article is also available on YouTube!

The Token Bucket algorithm is a flexible and efficient rate-limiting mechanism.

It works by filling a bucket with tokens at a fixed rate (e.g., one token per second).

Each request consumes a token, and if no tokens are available, the request is rejected.

The bucket has a maximum capacity, so it can handle bursts of traffic as long as the burst doesn’t exceed the number of tokens in the bucket.

Fixed Window Counter Rate Limiter (Redis & Java)

Mon, 20 Jan 2025 16:34:18 +0000

This article is also available on YouTube! The Fixed Window Counter is the simplest and most straightforward rate-limiting algorithm. It divides time into fixed intervals (e.g., seconds, minutes, or hours) and counts the number of requests within each interval. If the count exceeds a predefined threshold, the requests are rejected until the next interval begins. Looking for a more precise algorithm? Take a look at the Sliding Window Log implementation .

Rate limiting with Redis: An essential guide

Mon, 13 Jan 2025 18:23:26 +0000

This article is also available on YouTube! Rate limiting — it’s something you’ve likely encountered, even if you haven’t directly implemented one. For example, have you ever been greeted by a “429 Too Many Requests” error? That’s a rate limiter in action, protecting a resource from overload. Or maybe you’ve used a service with explicit request quotas based on your payment tier — same concept, just more transparent. Rate limiting isn’t just about setting limits; it serves a variety of purposes. Take Figma, for instance. Their rate limiter, built with Redis, saved them from a spam attack where bad actors sent massive document invitations to random email addresses. Without it, Figma could have faced skyrocketing email delivery costs and damaged reputation. Or look at Stripe: as their platform grew, they realized they couldn’t just throw more infrastructure at the problem. They needed a smarter solution to prevent resource monopolization by misconfigured scripts or bad actors. These stories show just how versatile rate limiting is. It prevents abuse, ensures fair access, manages load, cuts costs, and even protects against downtime. But here’s the kicker: the hard part isn’t knowing why you need a rate limiter. The real challenge is building one that’s both efficient and tailored to your needs.

SQL Best Practices Every Java Engineer Must Know

Tue, 01 Oct 2024 17:52:46 +0000

In the world of software development, best practices for SQL optimization are a critical skill that every Java engineer must master. Efficient database interactions can significantly enhance the performance of your applications, leading to faster response times and a better user experience.

This comprehensive guide dives into the best practices for SQL query optimization, tailored specifically for Java engineers. From understanding the importance of indexes to mastering joins and leveraging connection pooling, this article covers all the essential techniques you need to write efficient and performant SQL queries and best practices for SQL.

Foojay Podcast #56: Vectors in Java Code, Databases, and LLMs

Mon, 08 Jul 2024 07:33:28 +0000

In this Foojay podcast, we enter the world of mathematics by discussing Vectors and how they are crucial for AI and machine learning.

As ChatGPT explains: “A Vector is a mathematical structure that holds numerical values. Vectors are fundamental to the field of Artificial Intelligence, as they allow mathematical operations to be performed efficiently and form the basis of many machine learning algorithms.”

OK, but… how are these vectors crucial for the whole Artificial Intelligence evolution?

Exploring Cost-Effective Solutions for Stateful Rest Services

Mon, 27 May 2024 13:13:02 +0000

A Case Study with IBM Cloud Code Engine, EclipseStore, and IBM Cloud Object Storage

In the dynamic realm of cloud computing, finding cost-effective solutions for stateful rest services that don’t necessarily require scaling can be a nuanced challenge.

In this article, we’ll take a pragmatic look at a use case leveraging IBM Cloud Code Engine , EclipseStore , and IBM Cloud Object Storage (COS) . The goal is to shed light on how this can be a practical and economical alternative, particularly when scalability is not a primary concern.

Minimize costs utilizing the cloud with Spring-Data-Eclipse-Store

Tue, 13 Feb 2024 15:26:37 +0000

Sooner or later any Spring application needs to store data. And of course, the first and easiest move is to utilize Spring Data JPA . You can set up your data storage without knowing which relational database you will use in production and simply start coding without worrying about that.

It is the easiest and most convenient way to store data (in a relational database like PostgreSQL), yet in the cloud environment it is also an expensive way. Pricing at AWS RDS starts at 30$/month with tiny-Instances and always have fixed monthly costs regardless of its usage.

MariaDB Java Connector, Benchmark, and Java 21 virtual threads

Tue, 23 Jan 2024 13:03:18 +0000

Why should Java and database developers care about Virtual Threads? Developers can write faster and more resource efficient applications without having to refactor synchronous code a.k.a. “imperative”) JDBC code into asynchronous (a.k.a “reactive”) code that is hard to create, debug and maintain.

Writing scalable network code is always difficult. Synchronous APIs can’t be scaled beyond a certain point because they can block when performing I/O operations, which in turn block the thread until the operation is ready. For example, if you try to read data from a socket when no data is available, the thread will freeze until data becomes available. Threads are also an expensive resource on the Java platform, so we don’t want to force them to wait for I/O operations to complete. To avoid these limitations, we typically use asynchronous or reactive I/O frameworks such as Project Reactor, RxJava, or Vert.x and connectors like R2DBC. These frameworks allow us to write code that doesn’t block a thread during an I/O operation, but instead uses a callback or event notification to notify the code when the I/O operation is complete or ready.

Patterns For The Design Of Microservices – Part 2

Mon, 30 Oct 2023 08:58:06 +0000

In the previous article , we discussed some of the design patterns employed in the creation of microservices. In this subsequent article, we will delve into the remaining patterns that are commonly utilized in the realm of microservices.

Database Patterns

There are seven distinct patterns that further categorize the data management.

Database per Service
Shared Database
Saga Pattern
Command-Query Responsiblity Segregation (CQRS)
API Composition
Domain Event
Event Sourcing

In the next section, let’s discuss each pattern individually.

Reactive Database Access on the JVM

Mon, 17 Jul 2023 09:18:37 +0000

A couple of years ago, Reactive Programming was all the rage, but it had one big issue: reactive stopped as soon as you accessed a SQL database.

You had a nice reactive chain up to the database, defeating the whole purpose. Given the prevalence of SQL databases in existing and new apps, one couldn’t enjoy the full benefits of Reactive Programming but still pay the full price of complexity.

Since then, the landscape has changed tremendously. Most importantly, it offers many reactive drivers over popular databases: PostgreSQL, MariaDB and MySQL, Microsoft SQL Server, Oracle, you name it!

Journeys in Java, Level 10: Service Discovery with Eureka

Thu, 13 Jul 2023 06:39:06 +0000

Gaining complexity in a microservices system certainly isn’t for the faint of heart (though neither is complexity in monoliths!).

When there are many services that need to communicate with one another, we might need to coordinate multiple services communicating with multiple other services.

We also might code for varying environments such as local, development server, or the cloud.

How do services know where to find one another?
How can we avoid problems when a service is unavailable?
How do we handle requests when we scale up or down certain parts of our system?

This is where something like Spring Cloud Netflix Eureka comes into play.

A Dissection of Java JDBC to PostgreSQL, Part 2: Batching

Tue, 04 Jul 2023 13:56:41 +0000

This is the second part of a series where I look into how Java JDBC connections to PostgreSQL are working, to understand how to optimally implement them.

If you’ve landed on this article, it might be a good idea to read my first article as well, which gives an introduction and shares how JDBC communicates with PostgreSQL using simple and extended protocols.

This new article specifically explores batching. Batching for PostgreSQL JDBC is a property of the extended protocol. It uniquely allows you to send multiple statements in a single request.

Neo4j's Online Conference Is Coming In October...

Wed, 07 Jun 2023 08:46:15 +0000

NODES is back for its fifth year on October 26, 2023!

It’s the biggest graph community gathering dedicated to learning how to integrate graph technologies into ML and dev projects.

For 24 hours nonstop, speakers from around the world will showcase their implementations, tools, models, and more – sharing everything they’ve learned over this past year.

Join thousands of fellow developers, data scientists, and data engineers to get firsthand knowledge from peers who will show you the ropes – for free! So be sure to save your seat !

Managing Data Residency: The Demo

Mon, 29 May 2023 07:33:28 +0000

I explained the concepts and theory behind Data Residency in a previous post .

It’s time to get our hands dirty and implement it in a simple demo.

The sample architecture

In the last section of the previous post, I proposed a sample architecture where location-based routing happened at two different stages:

The API Gateway checks for an existing X-Country header. Depending on its value, it forwards the request to the computed upstream; If no value is found or no value matches, it forwards it to a default upstream.
The application uses Apache Shardingsphre to route again depending on the data. If the value computed by the API Gateway is correct, the flow stays “in its lane”; if not, it’s routed to the correct database, but with a performance penalty as it’s outside its lane.

I simplified some aspects:

A Dissection of Java JDBC to PostgreSQL Connections

Fri, 12 May 2023 08:52:52 +0000

Many Java applications use a database, and many use JDBC and PostgreSQL .

The latency of database usage is often important, yet, what exactly happens at the database level is unknown.

This article explains which options are available, what happens, and why.

Most importantly, it includes a small Java class that can be used and modified to execute specific database tasks, plus a description of how to use a utility (Wireshark ) to see what is actually happening.

Creating Scalable OpenAI GPT Applications in Java

Thu, 27 Apr 2023 14:18:45 +0000

One of the more notable aspects of ChatGPT is its engine, which not only powers the web-based chatbot but can also be integrated into your Java applications.

Whether you prefer reading or watching, let’s review how to start using the OpenAI GPT engine in your Java projects in a scalable way, by sending prompts to the engine only when necessary:

Imagine you want to visit a city and have a specific budget in mind. How should you spend the money and make your trip memorable? This is an excellent question to delegate to the OpenAI engine.

Ingesting Data from Kafka into the QuestDB time-series database

Tue, 28 Mar 2023 09:38:35 +0000

If you are working on a project with fast or streaming data, chances are Apache Kafka is already part of your pipeline .

But if you want to analyse your data, you will need to ingest from Kafka into some destination.

For time-series data, QuestDB can be an excellent choice.

For those of you who are not yet familiar with QuestDB , it is an Apache 2.0 licensed database designed for high throughput ingestion and fast SQL queries.

5 Great Reasons to use jOOQ

Wed, 22 Mar 2023 14:45:13 +0000

Hi, I’m Lukas . I was invited to talk about the business behind jOOQ on Foojay.io and as a short introduction to jOOQ , I’d like to highlight 5 great reasons to use it.

Before discussing those reasons:

What is jOOQ

jOOQ is an internal domain-specific language (DSL) modelling the SQL language as an API directly in Java.

This provides compile-time type safety to your query and enables a lot of other interesting features that I’ll show later.

How to Best Use Java Records as DTOs in Spring Boot 3

Thu, 09 Mar 2023 17:06:46 +0000

With the Spring 6 and Spring Boot 3 releases, Java 17+ became the baseline framework version.

So now is a great time to start using compact Java Records as Data Transfer Objects (DTOs ) for various database and API calls.

Whether you prefer reading or watching, let’s review a few approaches for using Java records as DTOs that apply to Spring Boot 3 with Hibernate 6 as the persistence provider.

Journeys in Java, Level 9: Docker compose all the things

Sun, 26 Feb 2023 10:02:15 +0000

Our microservices project contains quite a few pieces now.

We have two databases, three API services, a user-view service for books, and a service to host our configuration.

With so many pieces to manage, it would be nice to have something that orchestrates the individual services into a system, such as Docker Compose.

Back in our Level 5 rendition, we did exactly this for our smaller version of the project.

Now that we have expanded our services, we need to add those new pieces into the existing Docker Compose management umbrella.

Journeys in Java, Level 8: Add MongoDB to Spring Cloud Config

Thu, 09 Feb 2023 16:56:28 +0000

In our last article , we used Spring Cloud Config to provide database credentials to a microservice application connecting to a cloud-hosted Neo4j database. This post will backport this concept to our existing MongoDB database instance and its related microservices.

We will add our MongoDB credentials to the config server, so that it will be the central place for both our Neo4j and MongoDB database access. However, each service only has access to the credentials that it needs to operate, which provides some level of security through “separation of concerns” (versus universal access).

Journeys in Java, Level 7: Externalize Microservice Configuration

Mon, 30 Jan 2023 15:43:16 +0000

Our decided next step for this project takes us in a new direction by adding externalized configuration for our applications.

We already saw how to do something similar with MongoDB database credentials in Docker Compose to access a local database container, but what do we do when the database is hosted on public cloud or contains sensitive data?

We probably don’t want to publish our credentials anywhere or even prevent accidental publishing.

How Java Litters Beyond the Heap: Solid-State Drives

Fri, 27 Jan 2023 10:54:47 +0000

A Java application dutifully executes your logic, leaving behind footprints in the Java heap in the form of dead objects.

A garbage collector will then step in and clean out the memory for the new data. This cycle repeats until the app is stopped. This is well known.

But, the Java heap is one of many places where your app can generate garbage.

The application can also litter other parts of the software stack. It’s not done deliberately, but because some stack components also take advantage of garbage collection.

How Java Litters Beyond the Heap: Distributed Databases

Tue, 20 Dec 2022 11:40:43 +0000

Let’s create a simple Java application using a distributed database for the user data and see how the database generates litter in response to application requests.

While the Java Virtual Machine executes our code at runtime, it generates garbage in the heap, which is dutifully collected by the garbage collector. This is a known fact.

However, at the same time, Java apps don’t live in isolation and frequently communicate with services, databases, and other components. Some of those components also generate and collect garbage, while fulfilling the Java app’s requests. This is less known.

How Java Litters Beyond the Heap: Relational Databases

Tue, 29 Nov 2022 19:08:04 +0000

As Java developers, we’re no strangers to the concept of garbage collection. Our apps generate garbage all the time, and that garbage is meticulously cleaned out by CMS, G1, Azul C4, and other types of collectors.

However, the story doesn’t end with the Java heap. In fact, it is just the beginning.

In this article, we will create a simple Java application that uses a relational database for the user data and solid-state drives (SSDs) as a storage device. From here, we’ll explore how the application generates garbage at the database and SSD levels, while executing the application logic.

A Poor Man's API

Tue, 29 Nov 2022 09:13:38 +0000

Creating a full-fledged API requires resources, both time and money.

You need to think about the model, the design, the REST principles, etc., without writing a single line of code.

Most of the time, you don’t know whether it’s worth it: you’d like to offer a Minimum Viable Product and iterate from there.

I want to show how you can achieve it without writing a single line of code.

The solution

The main requirement of the solution is to use the PostgreSQL database . It’s a well-established Open Source SQL database.

Unified Event-Driven Architecture for the Cloud Native Enterprise

Thu, 06 Oct 2022 19:58:06 +0000

CTOs and enterprise architects have long recognized the importance of event-driven architectures (EDA). While once considered purely a technology concern, the foresight of organizations that have invested in EDA has become readily apparent as the world has shifted around us. In the past decade, we’ve witnessed changes in nearly every aspect of our technological worlds, and the vast majority of those have been affected in some way by a move toward event-driven, real-time processing.

Reclaiming Persistent Volumes in Kubernetes

Wed, 28 Sep 2022 09:55:19 +0000

Kubernetes is a widely used open-source container management platform for running stateless, containerized applications at scale. In recent years, Kubernetes has been extended to also support stateful workloads, including databases and key-value stores.{#43de}

There are three important API resources when it comes to managing stateful applications in Kubernetes:{#8e8a}

STSs schedule stateful pods, which can claim PVs through PVCs and mount them as volumes. Once a PV is claimed by an STS replica, Kubernetes will make sure that the volume stays with the replica, even if the pod gets rescheduled.{#e855}

How the world caught up with Apache Cassandra

Thu, 15 Sep 2022 16:50:50 +0000

The O’Reilly book, Cassandra: The Definitive Guide, features a quote from Ray Kurzweil, the noted inventor and futurist: {#speakable-summary}

“An invention has to make sense in the world in which it is finished, not the world in which it is started.”

This quote has a prophetic ring to it, especially considering my co-author Eben Hewitt included it in the 2010 first edition of this book we wrote, back when Apache Cassandra, the open-source, distributed, and highly scalable NoSQL database, was just on its 0.7 release.

Baeldung Series Part 2: Build a Dashboard With Cassandra, Astra and CQL – Mapping Event Data

Tue, 13 Sep 2022 17:46:02 +0000

1. Introduction

In our previous article , we looked at augmenting our dashboard to store and display individual events from the Avengers using DataStax Astra , a serverless DBaaS powered by Apache Cassandra using Stargate to offer additional APIs for working with it.

In this article, we will be making use of the exact same data in a different way. We are going to allow the user to select which of the Avengers to display, the time period of interest, and then display these events on an interactive map. Unlike in the previous article, this will allow the user to see the data interacting with each other in both geography and time.

Multi-cluster Cassandra with Google Kubernetes Engine (Pt. 2)

Thu, 08 Sep 2022 17:20:08 +0000

This is the second in a series of posts examining patterns for using K8ssandra to create Cassandra clusters with different deployment topologies.

In the first article in this series, we looked at how you could create a Cassandra cluster with two datacenters in a single cloud region, using separate Kubernetes namespaces in order to isolate workloads. For example, you might want to create a secondary Cassandra datacenter to isolate a read-heavy analytics workload from the datacenter supporting your main application.

Running your Database on OpenShift and CodeReady Containers

Tue, 06 Sep 2022 09:43:57 +0000

Let’s take an introductory run-through of setting up your database on OpenShift, using your own hardware and RedHat’s CodeReady Containers.

CodeReady Containers is a great way to run OpenShift K8s locally, ideal for development and testing. The steps in this blog post will require a machine, laptop or desktop, of decent capability; preferably quad CPUs and 16GB+ RAM.

Download and Install RedHat’s CodeReady Containers

Download and install RedHat’s CodeReady Containers as described in Red Hat OpenShift 4 on your laptop: Introducing Red Hat CodeReady Containers

Faster Integration Tests with Reusable Testcontainers

Wed, 17 Aug 2022 11:05:05 +0000

In my job, I often need integration tests against a Neo4j database. My software is written in Java, and so is Neo4j. Neo4j is embeddable, too. So, in theory, I could just depend on the embedded version, open a connection, test my stuff and call it a day. It would not be a different engine, it would be - at least from a query-engine and planer perspective - the same thing, in contrast to using an embedded SQL database as a drop-in for PostgresQL for example.

Deploy a Multi-Datacenter Cassandra Cluster in Kubernetes

Wed, 10 Aug 2022 08:27:59 +0000

The Get Started examples on the K8ssandra site are primarily concerned with spinning up a single Apache Cassandra datacenter in a single Kubernetes cluster.

However, there are many situations that can benefit from other deployment options.

In this series of articles, we’ll examine different deployment patterns and show how to implement them using K8ssandra.

Flexible topologies with Cassandra

From its earliest days, Cassandra has included the ability to assign nodes to datacenters and racks. A rack was originally conceived as mapping to a single rack of servers connected to shared resources, like power, network, and cooling.

Native GraphQL API with Neo4j AuraDB on Heroku

Mon, 08 Aug 2022 07:29:38 +0000

In my first article on Foojay, I would like to present one of many possible approaches to create a GraphQL API. I work at Neo4j , so it should not be a big suprise that I will use the Graph database with the same name as a backend for the application.

In this post I will cover a couple of things and I’ll start with a simple use case. Conceptually we will look at the “Schema-First” vs “Object-First” discussions and why - at least in my opinion - GraphQL can be seen much more like an object mapping concept than a query language itself. You will learn that while Neo4j is a Graph database, it does not have a built-in GraphQL layer. It does however a great query language called “Cypher” .

Five Data Models for IoT: Managing the Latest IoT Events Based on a State in Apache Cassandra

Thu, 28 Jul 2022 17:36:28 +0000

Apache Cassandra is a rock-solid choice for managing IoT and time series data at scale. The most popular use case of storing, querying and analyzing time series generated by IoT devices in Cassandra is well-understood and documented. In general, a time series is stored and queried based on its source IoT device. However, there exists another class of IoT applications that require quick access to the most recent data generated by a collection of IoT devices based on a known state. The question that such applications need to answer is: Which IoT devices or sensors are currently reporting a specific state? In this blog post, we focus on this question and provide five possible data modeling solutions to efficiently answer it in Cassandra.

NoSQL Use Cases: When to Use a Non-Relational Database

Thu, 28 Jul 2022 07:32:07 +0000

For decades, many companies have relied on relational databases to store, protect, and access their data.{#d8cf}

SQL databases, in particular, worked well for a long time and still do for many use cases. But, today, there is a wide range of situations where SQL databases can no longer satisfy the needs of modern enterprises, especially those that have made the move to the cloud.{#d8cf}

Increasingly, these companies are turning to NoSQL databases to meet their goals.{#d8cf}

Kubernetes and the Data Centre: Is Full Scale Migration Possible?

Thu, 21 Jul 2022 13:48:13 +0000

Data centres used to be exactly that - the centres of our data worlds, where all information, files and records would be kept and controlled.

Today, that is no longer the case. More companies are moving to cloud services, whether this is dialling down on their traditional on-premises environments or never setting them up in the first place.

This is not to say that data centres will be wholly abandoned for cloud, but the mix will continue to evolve and be very different to today. Gartner predicts that by 2025, 85% of infrastructure strategies will take a more hybrid approach, integrating on-premises, colocation, cloud and edge delivery options, compared with 20% in 2020.

Why a Cloud-Native Database Must Run on K8s

Tue, 19 Jul 2022 17:44:09 +0000

We’ve been talking about migrating workloads to the cloud for a long time, but a look at the application portfolios of many IT organizations demonstrates that there’s still a lot of work to be done. In many cases, challenges with persisting and moving data in clouds continue to be the key limiting factor slowing cloud adoption, despite the fact that databases in the cloud have been available for years.

For this reason, there has been a surge of recent interest in data infrastructure that is designed to take maximum advantage of the benefits that cloud computing provides. A cloud-native database is one that achieves the goals of scalability, elasticity, resiliency, observability and automation; the K8ssandra project is a great example. It packages Apache Cassandra and supporting tools into a production-ready Kubernetes deployment.

Journeys in Java, Level 6: Build a Neo4j Microservice

Wed, 13 Jul 2022 15:13:43 +0000

For our next adventure in Java microservices, we want to expand our system for book review data.

While books and authors are well-suited for a document database such as MongoDB, once you add reviews to the mix, the importance of the relationships makes this project better suited for a graph database.

This way, we can utilize relationships between the different entities to improve analysis based on the structure of the connections.

The End of the Beginning for Apache Cassandra

Tue, 12 Jul 2022 17:43:39 +0000

Image: Pixabay

Editor’s note: This story originally ran on July 27, 2021, the day that Apache Cassandra 4.0 was released.{#4bbb}

Today is a big day for those of us in the Apache Cassandra community. After a long uphill climb, Apache Cassandra 4.0 has finally shipped . I say finally, because it has at times seemed like an elusive goal. I’ve been involved in the Cassandra project for almost 10 years now and I have seen a lot of ups and downs.{#d6a4}

MicroStream - Part 5: Caching, Integrations and Clustering

Wed, 06 Jul 2022 07:11:38 +0000

In this last article of the series, we cover a few other MicroStream features: caching, clustering, and the integrations into other frameworks.

We have now covered three main aspects of MicroStream:

We have seen how you can configure the StorageManager to define where data is stored.
We have discussed the strategy that you should follow to make sure the changes are also reflected in the storage so that we do not lose data.
We described the serialisation framework of MicroStream that stores the objects in a binary format so that it survives a process restart.

In this last article of the series, we cover a few other features of MicroStream: caching, integrations into other products, and we discuss a major upcoming feature.

MicroStream Next-Generation Serialisation Engine

Wed, 29 Jun 2022 08:07:20 +0000

In this fourth part we go deeper into the Serialisation engine that is within MicroStream to store the Object graph in a binary format.

In the previous articles (part 1 , part 2 and part 3 ), we have already mentioned that MicroStream stores Java instances in storage in a binary way with a new, from the ground up created, serialisation framework.

In this article, we go a bit more in detail about the next generation Java serialisation that we have built to achieve the MicroStream Java Object database and how you can use it outside the functionality of storing the root object that makes up your database.

K8ssandra ramps up security features to match Kubernetes’ best practices

Thu, 23 Jun 2022 19:54:58 +0000

New security features are coming to the open-source data platform: K8ssandra. The goal? To align even more with the security best practices of Kubernetes. Here’s an introduction to the platform’s security mission and an update on current initiatives.{#e3b8}

The security defaults applied by K8ssandra are about to get even more aligned withKubernetes’ security practices . In an upcoming release of K8ssandra, pod and container security configurations give users full customization capabilities and default values out of the box.{#c373}

MicroStream: How to store your data?

Wed, 22 Jun 2022 11:10:55 +0000

In the third article of the MicroStream series, we go into the details what you need to do so that data is stored externally to survive the process restart.

In the previous articles in this series, we introduced the main features of the framework and how you can configure the StorageManager .

Java instances in memory are your database but the StorageManager makes sure that the data are persisted so that they are read the next time the runtime is started again.

Data Modeling in Cassandra and Astra DB

Thu, 16 Jun 2022 19:22:05 +0000

What does it take to build an efficient and sound data model for Apache Cassandra ® and DataStax Astra DB ? Where would one start? Are there any data modeling rules to follow? Can it be done consistently time and time again? The answers to these and many other questions can be found in the Cassandra data modeling methodology.{#3856}

In this post, we present a high-level overview of the data modeling methodology for Cassandra and Astra DB , and share over half a dozen complete data modeling examples from various real-life domains. We apply the methodology to create Cassandra and Astra DB data models for IoT, messaging data, digital library, investment portfolio, time series, shopping cart, and order management. We even provide our datasets and queries for you to try.{#4bd9}

MicroStream: Configure the StorageManager

Wed, 15 Jun 2022 08:04:11 +0000

In this second article in the series, we cover how to get started configuring the StorageManager of MicroStream!

As we have discussed in the introduction article of this series, MicroStream provides you with a solution to use Java instances in memory as your database.

You can access the data through getters, the Stream API, or any other method provided on the Java Objects.

This means ultrafast, no mapping required, and no external system for your data.

Tombstones and Ghost Data Don’t Have to Be Scary!

Thu, 09 Jun 2022 20:36:25 +0000

Working with tombstones can be one of the more frustrating aspects of Apache Cassandra®. Problems with tombstones can happen when a large number of deletes occur within a short period of time, resulting in slower queries and disk bloat. This article will discuss tombstones, including the purpose they serve, and strategies/methods on how to avoid them.{#2508}

Tombstones in Apache Cassandra ® are written markers that are created whenever a value is deleted. Their purpose is to indicate that any prior values have been deleted. When a read occurs, it pulls data from the sorted string table (SSTable) files on a node that may contain the data. All prior values for the requested key are returned, ordered by write-time.{#d711}

Managing Distributed Applications in Kubernetes Using Cilium and Istio with Helm and Operator for Deployment

Tue, 07 Jun 2022 14:51:03 +0000

This post will show you the benefits of managing your distributed applications with Kubernetes in cross-cloud, multi-cloud, and hybrid cloud scenarios using Cilium and Istio with Helm and Operator for deployment.{#6864}

In our recent post on The New Stack , we showed you how you can leverage Kubernetes (K8s) and Apache Cassandra TM to manage distributed applications at scale, with thousands of nodes across both on-premises and in the cloud. In that example, we used K8ssandra and Google Cloud Platform (GCP) to illustrate some of the challenges you might expect to encounter as you grow into a multi-cloud environment, upgrade to another K8s version, or begin working with different distributions and complimentary tooling. In this post, we’ll explore a few alternative approaches to using K8s to help you more easily manage distributed applications.{#fec3}

MicroStream: High-Performance Java-Native-Persistence framework

Mon, 06 Jun 2022 08:03:02 +0000

Within every application, data plays a very important role.

In every industry, you have data, such as customer or product information, analysis results, patient data, or account data is a vital part of the business and the applications that support your business processes.

Data must be in memory to be processed by your application, but must be stored outside of it because the application’s memory is temporary.

Information is lost when the process ends unless we have stored it somewhere else in a sustainable way.

How DataStax Tracked Down a Linux Kernel Bug with Fallout

Thu, 02 Jun 2022 15:32:08 +0000

Sometimes as a developer, you run into a bug buried deep within the layers of your software stack. Chasing down the root cause requires not only curiosity, patience, and a healthy dose of tenacity but a willingness to try different tools and approaches. This post describes our challenges and ultimate success in tracking down a Linux kernel bug using Fallout.{#ee66}

Bugs come in all shapes and sizes and it’s not always clear at the beginning of a debugging session which one you’re currently chasing. Some bugs can be fixed up in a matter of minutes while others take weeks to nail down. And the really tricky ones require you to dig through multiple layers of your software stack, stressing the limits of your patience.{#8aae}

Journeys in Java, Level 5: Building an Empire of Microservices

Thu, 02 Jun 2022 05:08:28 +0000

We continue building our microservices system by adding a coordination layer to handle spinning multiple services up and down.

In previous articles, we grew from two Spring Boot applications to three applications + a database container.

As we continue to scale and broaden our system, we will need something to coordinate service and ensure consistent communication (avoid human error).

We will do this through docker-compose , an orchestration tool that manages containerized applications.

Optimizing Distributed Joins: The Case of Google Cloud Spanner and DataStax Astra DB

Tue, 31 May 2022 16:25:25 +0000

In this post, learn how relational and NoSQL databases, Google Cloud Spanner and DataStax Astra DB, optimize distributed joins for real-time applications.{#c63a}

Distributed joins are commonly considered to be too expensive to use for real-time transaction processing. That is because, besides joining data, they also frequently require moving or shuffling data between nodes in a cluster, which can significantly affect query response times and database throughput. However, there are certain optimizations that can completely eliminate the need to move data to enable faster joins. In this article, we first review the four types of distributed joins, including shuffle join, broadcast join, co-located join, and pre-computed join. We then demonstrate how leading fully managed Relational and NoSQL databases, namely Google Cloud Spanner and DataStax Astra DB , support optimized joins that are suitable for real-time applications.{#bea1}

Why Developers Should Use Apache Pulsar

Thu, 19 May 2022 15:10:49 +0000

Apache Pulsar is an open source streaming platform that addresses some important limitations in Kafka, particularly for cloud-native applications.
SPainter VFX / Getty Images

If you are building applications today, you are probably familiar with the microservices model: Rather than building big monolithic applications, we break services down into isolated components that we can independently update or change over time.

Microservices deployments then can use a message bus to decouple and manage the communication between services, which makes it easier to replay requests, handle errors, and deal with load spikes and rapid increases in requests while maintaining the serialized order.

Bring Streaming to Apache Cassandra with Apache Pulsar

Thu, 05 May 2022 18:10:42 +0000

Twitch, YouTube, Instagram, Facebook — virtually every major brand nowadays uses live streaming to connect and engage their audience.{#2544}

For enterprises and developers building cloud-native applications , this growing trend creates a need for streaming technologies that can reliably handle the rush of massive amounts of data, while also being flexible and easy to manage for developers.{#2544}

One such technology is Apache Pulsar ® — an open-source, distributed messaging and streaming platform that’s easy to deploy, simple to scale, and packed with developer-friendly APIs.{#8b5a}

Journeys in Java, Level 4: Building an Empire of Microservices

Wed, 27 Apr 2022 07:22:03 +0000

This article is the fourth iteration in a series of posts about building microservices in Java.

The initial post started with a single string message passed between two locally-running applications, and we are gradually adding services and functionality along the way.

In this level 4 instalment, we will add a new service that will manage book authors. It will call a database containing a data domain of books and retrieve authors and their information. This will entail creating a new, separate service from our previously written service1 and service2.

SQL vs. NoSQL: Pros & Cons

Thu, 21 Apr 2022 09:28:50 +0000

What’s the best way to store, protect and access your data?{#abb7}

This is a fundamental, yet critical decision.{#abb7}

After all, data is the cornerstone of success for just about every modern organization.{#abb7}

For most companies, the choice comes down to SQL and NoSQL databases. Each has unique strengths and weaknesses.{#abb7}

SQL databases have been a proven option since the 1970s. They are made up of highly structured tables, consisting of rows and columns, related to one other through common attributes. Every column is required to have a value for its corresponding row.

NoSQL (“not only SQL” or “non-SQL”) databases came along later to break the relational table straitjacket, with the ability to store and access all data types, structured and unstructured, together.

They’re extremely flexible and easy for developers to work with and modify. Learn more about SQL and NoSQL databases and their basic differences .{#3ad0}

Have you upgraded to Cassandra 4.0 yet?

Fri, 11 Mar 2022 08:35:20 +0000

Alongside stability, Apache Cassandra 4.0 stands out for its ability to scale operations faster, its new auditing capabilities, and the way it embraces privacy by design. In this article, we’ll cover the new features and walk you through the steps required to upgrade to Cassandra 4.0.

The general availability of Apache Cassandra 4.0 marks the most stable release in the project’s history.

Even with the 4.0 beta 1 release back in July 2020, Project Management Committee (PMC) members were confident the latest version of the distributed NoSQL database would be ready to go to work with Apache Cassandra Committer; PMC member Sankalp Kohli at the time indicated that “users will be able to use the release knowing it is production-ready on day one.”

Journeys in Java, Level 3: Building an Empire of Microservices

Fri, 04 Mar 2022 16:23:40 +0000

We are still in the beginning stages of building microservices with Java, but we have already seen how to send and receive communications between two Spring Boot applications. In the first blog post of this series, our two applications transmitted a message string of "Hello, World!". In the second blog post , we took it one step further by embedding a MongoDB database into our service1, added a data domain (books) with four pre-populated entries, and transmitted the Book information between the two applications.

The Serverless Database You Really Want

Thu, 03 Mar 2022 20:43:05 +0000

The dreaded part of every site reliability engineer’s (SRE) job eventually: capacity planning. You know, the dance between all the stakeholders when deploying your applications. Did engineering really simulate the right load and do we understand how the application scales? Did product managers accurately estimate the amount of usage? Did we make architectural decisions that will keep us from meeting our SLA goals? And then the question that everyone will have to answer eventually: how much is this going to cost? This forces SREs to assume the roles of engineer, accountant and fortune teller.

Why Pulsar Beats Kafka for a Scalable, Distributed Data Architecture

Thu, 03 Mar 2022 14:10:45 +0000

The leading open source event streaming platforms are Apache Kafka and Apache Pulsar. For enterprise architects and application developers, choosing the right event streaming approach is critical, as these technologies will help their apps scale up around data to support operations in production.

Everyone wants results faster. We want applications that know what we want, even before we know ourselves. We want systems that constantly check for fraud or security issues to protect our data. We want applications that are smart enough to react and change plans when faced with the unexpected. And we want those services to be continuously available.

Save Hours on your Setup of K8ssandra with the New Config Builder

Thu, 24 Feb 2022 19:36:47 +0000

Setting up K8ssandra in your workflow just got a whole lot easier. With the new Config Builder you can be running Apache Cassandra® on Kubernetes in a matter of minutes.

The purpose of K8ssandra is to make it easy to run Apache Cassandra® on Kubernetes. We recently took another big step in that direction by releasing the K8ssandra config builder .

Even if you’ve created thousands of nodes or integrated K8ssandra in your stack, you’ll probably want to give the config builder a try. Walk through the interactive wizard defining the shape of your cluster, resource requirements, and toggle features to fit your needs. This is great for smoothing out the on-ramp for a production ready environment of K8ssandra.

Why we decided to build a K8ssandra Operator - Part 4

Tue, 22 Feb 2022 17:04:28 +0000

In the first , second , and third posts in this series, we’ve shared conversations with K8ssandra core team members on our journey to build a Kubernetes operator for K8ssandra. We’ve discussed the virtues of the Helm package manager versus Kubernetes operators for deploying and managing infrastructure in Kubernetes and some of our implementation choices for the operator.

In this final post of the series, we pick up from the previous post with a discussion of how we decided to structure our projects in GitHub, how we test the K8ssandra operator, and our hopes for how the operator will expand the K8ssandra developer community.

The Critical Role Streaming Plays in a Data Stack

Fri, 18 Feb 2022 17:20:13 +0000

Why Apache Pulsar is the right choice for multi-datacenter, geo-distributed deployments.

Building applications today requires companies and teams to be familiar with data flows (including streaming data flows) and how to transform them into compelling user experiences. Data fuels companies’ ability to understand their customers, move faster than competitors, and provide compelling products that the market wants. According to research by DataStax and Clearpath Strategies, almost all companies (96%) now have a formal strategy for how they will work with data, but only 38% think they’re doing a good job using data in their applications.

Journeys in Java, Level 2: Building an Empire of Microservices

Wed, 16 Feb 2022 15:29:33 +0000

Let’s dive into the world of microservices find out the complexities, best practices, and troubles. I will share all my learnings, as well!

In my last blog post , we began building microservices in Java with two Spring Boot applications, passing a “Hello, World!” string from one application to the other. We focused on reduced complexity, minimum previous knowledge, and few components. Next, we can slowly add pieces that simulate microservices projects in the real world.

Cassandra Database Migration to Kubernetes with Zero Downtime

Tue, 15 Feb 2022 15:22:15 +0000

K8ssandra is a cloud-native distribution of the Apache Cassandra® database that runs on Kubernetes, with a suite of tools to ease and automate operational tasks. In this post, we’ll walk you through a database migration from a Cassandra cluster running in AWS EC2 to a K8ssandra cluster running in Kubernetes on AWS EKS, with zero downtime.

As an Apache Cassandra user, your expectation should be that migrating to K8ssandra would happen without downtime. To make that happen with “classic” clusters running on virtual machines or bare metal instances, you will use the datacenter (DC) switch technique which is commonly used in the Cassandra community to transfer clusters to different hardware or environments. The good news is that it’s not very different for clusters running in Kubernetes as most Container Network Interfaces (CNI) will provide routable pod IPs.

Building Reactive Java Applications with Spring Framework

Tue, 15 Feb 2022 14:09:28 +0000

In one of our many free tutorials on DataStax Developers YouTube channel , we walked you through how to build a reactive implementation of Spring PetClinic in Apache Cassandra® using Spring WebFlux. The full series is available on YouTube .{#08a7}

If you’re a Java developer who uses the Spring ecosystem, you’ve probably seen the Spring Pet Clinic . In this workshop, we will walk you through a new reactive implementation of the Pet Clinic backend that uses Spring WebFlux and Apache Cassandra ® (via DataStax Astra DB ).{#47b5}

Building Scalable Streaming Applications with DataStax Astra Streaming

Wed, 09 Feb 2022 19:09:28 +0000

This post introduces developers to DataStax Astra Streaming with a step-by-step tutorial that illustrates how it can be used to easily build scalable, streaming applications.{#39ae}

Astra Streaming enables developers to build streaming applications on top of an elastically scalable, multi-cloud messaging and event streaming platform powered by Apache Pulsar . DataStax Astra Streaming is currently in beta, and we’ll be releasing a full demo soon. In the meantime, this article will walk you through a short demo that will provide a great starting point for familiarizing yourself with this powerful new streaming service.{#a6cf}

SQL vs. NoSQL: Pros & Cons

Wed, 09 Feb 2022 09:26:22 +0000

What’s the best way to store, protect and access your data? This is a fundamental, yet critical decision. After all, data is the cornerstone of success for just about every modern organization. For most companies, the choice comes down to SQL and NoSQL databases. Each has unique strengths and weaknesses.

SQL databases have been a proven option since the 1970s. They are made up of highly structured tables, consisting of rows and columns, related to one other through common attributes. Every column is required to have a value for its corresponding row. NoSQL (“not only SQL” or “non-SQL”) databases came along later to break the relational table straitjacket, with the ability to store and access all data types, structured and unstructured, together. They’re extremely flexible and easy for developers to work with and modify. Learn more about SQL and NoSQL databases and their basic differences .

Deploying to Multiple Kubernetes Clusters with the K8ssandra Operator

Fri, 04 Feb 2022 22:10:56 +0000

We built the new K8ssandra Operator to simplify deploying multiple Apache Cassandra data centers in different regions and across multiple Kubernetes (K8s) clusters. We’re not at the finish line just yet, but we’ve hit the first major milestone. Now, it’s easier than ever to run Apache Cassandra® across multiple K8s clusters in multiple regions with the K8ssandraCluster!

The K8ssandraCluster is a new custom resource for K8ssandra that covers all the bases necessary for installing a production-ready, multi-cluster K8ssandra deployment. Head over to the DataStax Tech blog to learn more about how to specify your remote clusters with the K8ssandraCluster, its deployment architecture, and what’s coming next in our continued development of the K8ssandra operator.

Backing up K8ssandra with MinIO

Thu, 03 Feb 2022 22:50:09 +0000

K8ssandra includes Medusa for Apache Cassandra® to handle backup and restore for your Cassandra nodes. Recently Medusa was upgraded to introduce support for all S3 compatible backends, including MinIO , the popular k8s-native object storage suite. Let’s see how to set up K8ssandra and MinIO to backup Cassandra in just a few steps.

Deploy MinIO

Similar to K8ssandra, MinIO can be simply deployed through Helm.

First, add the MinIO repository to your local list:

Unboxing K8ssandra: The Data Layer For Your Kubernetes-Powered Applications

Tue, 01 Feb 2022 17:51:35 +0000

A Complimentary Live Webinar, Sponsored by DataStax

Kubernetes made it easy to deploy and scale out your cloud-native applications. With K8ssandra , you can now scale application data with the same simplicity and high availability. Join us as we unbox K8ssandra a cloud native data layer for Kubernetes and explore how you can deploy it alongside your applications.

Install k8ssandra
Authenticate with Stargate
Query your data via a convenient API (REST, document, or graphql)

Not familiar with Cassandra? Astra DB is a great (free) place to learn without any of the infrastructure setup, or management headaches.

Taking Your Database Beyond a Single Kubernetes Cluster

Wed, 19 Jan 2022 10:31:47 +0000

By Christopher Bradford and Ty Morton {#31db}

Global applications need a data layer that is as distributed as the users they serve. Apache Cassandra has risen to this challenge, handling data needs for the likes of Apple, Netflix and Sony. Traditionally, managing data layers for a distributed application was handled with dedicated teams to manage the deployment and operations of thousands of nodes — both on-premises and in the cloud.

Kubernetes and Apache Cassandra: What Works (and What Doesn’t)

Fri, 14 Jan 2022 16:30:50 +0000

“I need it now and I need it reliable”

– ANYONE WHO HASN’T DEPLOYED APPLICATION INFRASTRUCTURE

If you’re on the receiving end of this statement, we understand you here in the K8ssandra community . Although we do have reason for hope. Recent surveys have shown that Kubernetes (K8s) is growing in popularity, not only because it’s powerful technology, but because it actually delivers on reducing the toil of deployment.

How to Put a Database in Kubernetes?

Thu, 13 Jan 2022 09:18:38 +0000

Learn the key steps of deploying databases and stateful workloads in Kubernetes and meet the cloud-native technologies, like K8ssandra, that can streamline Apache Cassandra for K8s.{#95cc}

The idea of running a stateful workload in Kubernetes (K8s) can be intimidating, especially if you haven’t done it before. How do you deploy a database? Where is the actual storage? How is the storage mapped to the database or the application using it?{#6cde}

Requirements for Running K8ssandra for Development

Thu, 06 Jan 2022 12:15:27 +0000

The K8ssandra Quick start is a excellent guide for doing a full installation of K8ssandra on a dev laptop and trying out the various components of the K8ssandra stack. While this is a great way to get your first hands-on experience with K8ssandra, let’s state the obvious: running K8ssandra locally on a dev laptop is not aimed at performance. In this blog post, we will start Apache Cassandra® locally then explain how to run benchmarks to help evaluate what level of performance (especially throughput) you can expect from a dev laptop deployment.

Why K8ssandra?

Thu, 06 Jan 2022 12:10:09 +0000

The problem: scalable data persistence for cloud-native applications

The past decade and a half has brought tremendous change in how we in the software industry think about building and delivering internet applications. With the launch of Amazon Web Services (AWS) in 2006, companies of any size could access cloud computing infrastructure. The burst of creativity that followed gave birth to concepts like cloud-native in 2010, and methodologies such as the twelve-factor app in 2011.

Simplify Migrating from Kafka to Pulsar with Kafka Connect Support

Wed, 05 Jan 2022 11:30:09 +0000

Large-scale implementations of any system, such as the event-streaming platform Apache Kafka, often involve customizations and tools and plugins developed in-house. When it’s time to transition from one system to another, the task can become complicated, drawn-out, and error-prone. Often the benefits of an alternative system (which can include significant cost savings and other efficiencies) are outweighed by the risks and costs of migration. As a result, an organization can end up locked into a suboptimal situation, footing a bigger bill than necessary and missing out on modern features that help move the business forward faster.

New Survey Finds Data on Kubernetes Is No Longer a Pipe Dream

Thu, 30 Dec 2021 15:52:00 +0000

For people that work in infrastructure and application development, the pace of change is quick. Finish one project and it’s on to the next. Each iteration requires an evaluation asking if the right technology is being used and if it provides a new advantage. Kubernetes has been on the fast track of continuous evaluation. New projects and methodologies are continuously emerging and it can be hard to keep up. Then there is the question of running stateful services. The Data on Kubernetes community has released a report titled “Data on Kubernetes 2021” to give us a snapshot of where our industry sits with stateful workloads. Over 500 executives and tech leaders were asked some very direct and insightful questions about how they use Kubernetes. It turns out that there were a lot of surprising finds. Some that I would have never predicted. Let’s dig into some of the highlights that stood out to me.

The future of cloud-native databases begins with Apache Cassandra 4.0

Wed, 29 Dec 2021 07:40:00 +0000

“Reliability at massive scale is one of the biggest challenges we face at Amazon.com, one of the largest e-commerce operations in the world; even the slightest outage has significant financial consequences and impacts customer trust.”

This was the first line of the highly impactful paper titled “Dynamo: Amazon’s Highly Available Key-value Store.” Published in 2007, it was written at a time when the status quo of database systems was not working for the massive explosion of internet-based applications. A team of computer engineers and scientists at Amazon completely re-thought the idea of data storage in terms of what would be needed for the future, with a firm footing in the computer science of the past. They were trying to solve an immediate problem but they had unwittingly sparked a huge revolution with distributed databases and the eventual collision with cloud-native applications.

Kubernetes Data Simplicity: Getting started with K8ssandra

Thu, 23 Dec 2021 08:56:22 +0000

You might have heard about the K8ssandra project and want to start contributing, or maybe you want to start using all of its features. If you aren’t familiar with K8ssandra (pronounced like “Kate Sandra”), you can read this overview before digging into the developer activities in this post.

In a nutshell, K8ssandra is an open-source distribution of Apache Cassandra™ for Kubernetes, which includes a rich set of trusted open-source services and tooling. K8ssandra comes with handy features that are baked-in and pluggable, which allows for flexible deployment and configuration.

Build a Status Dashboard Using Spring Boot and Astra DB

Fri, 17 Dec 2021 09:28:19 +0000

In this article, we are going to build “Tony Stark’s Avengers Status Dashboard”, used by The Avengers to monitor the status of the members of the team.

This will be built using DataStax Astra , a DBaaS powered by Apache Cassandra using Stargate to offer additional APIs for working with it.

On top of this, we will be using a Spring Boot application to render the dashboard and show what’s going on.

7 Reasons to Choose Apache Pulsar over Apache Kafka

Wed, 15 Dec 2021 10:03:13 +0000

I wrote an earlier version of this article in 2019, while I was CEO of Kesque, a real-time messaging service built on Apache Pulsar , the cloud-native distributed messaging and streaming platform. A lot of big changes have happened in the interim; perhaps the most significant of these is the fact that the company I founded in early 2019 was acquired , in January, by DataStax. One thing that hasn’t changed, however, is the rationale behind our choice of Apache Pulsar.

Adelphi: Apache Cassandra Testing Goes Cloud Native

Thu, 09 Dec 2021 15:55:43 +0000

|———————————————————————————————————————–| | adelphi (ancient greek ἀδελφός) - Subs.: brother or sister, son of the same mother. Adj.: double, twin, in pairs. |

What is Adelphi?

Adelphi is an open-source QA tool for Apache Cassandra™, it’s packaged as a Helm chart and it simplifies the tasks of running data integrity and performance tests on Kubernetes.

Built for simplicity, Adelphi is an opinionated tool; it runs an Argo workflow of well-defined steps and prefers convention over configuration, but it exposes some knobs that you can override.

Why a Serverless Data API Might be Your Next Database

Wed, 01 Dec 2021 14:53:44 +0000

App development stacks have been improving so rapidly and effectively that today there are a number of easy, straightforward paths to push code to production, on the cloud platform of your choice. But what use are applications without the data that users interact with? Persistent data is such an indispensable piece of the IT puzzle that it’s perhaps the reason the other pieces even exist.

Enter cloud and internet scale requirements, essentially mandating that back-end services must be independently scalable / modular subsystems to succeed. Traditionally, this requirement has been difficult in the extreme for stateful systems. No doubt, database as-a-service (DBaaS) has made provisioning, operations, and security easier. But as anyone who has tried to run databases on Kubernetes will tell you: auto scaling databases, especially ones that are easy for developers to use, remain out of reach for mere mortals.

Backing up K8ssandra with MinIO

Tue, 30 Nov 2021 17:00:00 +0000

Enterprise-Level Apache Cassandra Sink Connector for Apache Pulsar

Wed, 24 Nov 2021 19:16:16 +0000

When DataStax started investing in streaming with Apache Pulsar ™, we knew that one of the first things people would want to do was connect existing enterprise data sources to Apache Cassandra™ using Pulsar.

Apache Pulsar has a powerful framework called Pulsar IO to enable this kind of use case, and at DataStax we already had a best-in-class Kafka Connect Sink that enables you to store structured data coming from one or more Kafka topics into DataStax Enterprise, Apache Cassandra, and Astra.

K8ssandra: Production-Ready Platform for Cassandra on Kubernetes

Wed, 24 Nov 2021 09:45:12 +0000

K8ssandra , an open-source distribution of Apache Cassandra for Kubernetes, aims to provide a “production-ready platform”, and this includes automation for operational tasks such as repairs, backups, and monitoring. Cassandra is a distributed NoSQL database designed for global scale and fault tolerance for the most demanding applications on the planet, written in Java.

K8ssandra is deployed using Helm and supports stateful workloads out of the box, which enables database administrators (DBAs) and site reliability engineers (SREs) to set up and operate Cassandra clusters using best practices in a Kubernetes environment.

Connect Stateful Workloads Across Kubernetes Clusters

Fri, 23 Jul 2021 08:44:32 +0000

One of the biggest selling points of Apache Cassandra™ is its shared-nothing architecture, making it an ideal choice for deployments that span multiple physical datacenters. So when our Cassandra as-a-service single-region offering reached maturity, we naturally started looking into offering it cross-region and cross-cloud.

One of the biggest challenges in providing a solution that spans multiple regions and clouds is correctly configuring the network so that Cassandra nodes in different data centers can communicate with each other successfully, even as individual nodes are added, replaced, or removed. From the start of the cloud journey at DataStax, we selected Kubernetes as our orchestration platform, so our search for a networking solution started there. While we’ve benefited immensely from the ecosystem and have our share of war stories, this time we chose to forge our own path, landing on ad-hoc overlay virtual application networks (how’s that for a buzzword soup?).

The Search for a Cloud-Native Database

Fri, 16 Jul 2021 07:40:06 +0000

The concept of “cloud-native” has come to stand for a collection of best practices for application logic and infrastructure, including databases. However, many of the databases supporting our applications have been around for decades, before the cloud or cloud-native was a thing. The data gravity associated with these legacy solutions has limited our ability to move applications and workloads.

As we move to the cloud, how do we evolve our data storage approach? Do we need a cloud-native database? What would it even mean for a database to be cloud-native? Let’s take a look at these questions.

Best Practices for Data Pipeline Error Handling in Apache NiFi

Thu, 15 Jul 2021 05:25:32 +0000

Learn actionable strategies for error management modeling in Apache NiFi data pipelines, and understand the benefits of planning for error handling.

According to a McKinsey report, “the best analytics are worth nothing with bad data”. We as data engineers and developers know this simply as “garbage in, garbage out”. Today, with the success of the cloud, data sources are many and varied. Data pipelines help us to consolidate data from these different sources and work on it. However, we must ensure that the data used is of good quality. As data engineers, we mold data into the right shape, size, and type with high attention to detail.

A Case for Databases on Kubernetes from a Former Skeptic

Tue, 13 Jul 2021 09:12:18 +0000

Kubernetes is everywhere. Transactional apps, video streaming services and machine learning workloads are finding a home on this ever-growing platform. But what about databases?

If you had asked me this question five years ago, the answer would have been a resounding “No! " — based on my experience in development and operations. In the following years, as more resources emerged for stateful applications, my answer would have changed to “Maybe,” but always with a qualifier: “It’s fine for development or test environments…” or “If the rest of your tooling is Kubernetes-based, and you have extensive experience…”

Kubernetes Is The Best Technology For Running A Cloud-Native Database

Wed, 30 Jun 2021 09:13:19 +0000

For this reason, there has been a surge of recent interest in data infrastructure that is designed to take maximum advantage of the benefits that cloud computing provides. A cloud-native database is one that achieves the goals of scalability, elasticity, resiliency, observability, and automation; the K8ssandra project is a great example. It packages Apache Cassandra and supporting tools into a production-ready Kubernetes deployment.

Astra Service Broker: Tradeoff-Free Cassandra in Kubernetes

Thu, 19 Nov 2020 19:52:00 +0000

When you are designing and building a cloud-native application, you are probably thinking about deploying it with Kubernetes. What about the database? That can get a bit more complicated as you weigh out the tradeoffs of elasticity, scale, and self-healing requirements versus maintaining servers and the long term operations required. Apache Cassandra™ ticks the first three boxes easily, but what about the operational burden of managing Cassandra? That’s where the cloud-native Cassandra service, DataStax Astra , helps both operators and developers. But what makes it easy for deployments on Kubernetes to access the Astra managed service?