DataEngineering on foojay.io - Friends of OpenJDK

BoxLang 1.14.0 : Navigate Anything: JSONPath Comes to BoxLang's DataNavigator

Thu, 02 Jul 2026 18:17:16 +0000

Every application eventually has to deal with deeply nested data. JSON API responses with payloads six levels deep. Configuration files where the key you need is buried inside an array of objects, one of which has a null for the field you thought was required. Module metadata structures that nobody wrote a schema for. Runtime introspection data shaped like a tree that grew without a plan.

BoxLang introduced DataNavigators in version 1 so you can use dataNavigate(), call .from() to scope down, chain your .get() calls, provide defaults everywhere. Clean and safe. But even with the fluent API, extracting a specific slice of a complex payload still required multiple navigator hops, or a loop, or defensive key-exists checks stacked three levels deep.

JC-AI Newsletter #15

Fri, 20 Mar 2026 07:56:01 +0000

Over the past two weeks, the field of artificial intelligence has continued its remarkable pace of advancement. As AI becomes increasingly woven into the fabric of daily life, shaping how we work, communicate, and make decisions, it is both timely and valuable to step back and understand the broader trajectory of this technology. Whether the developments around us feel promising or challenging, one truth remains clear: AI is not simply leaving. It is here to stay, and understanding its evolution is essential from many perspectives.

JC-AI Newsletter #11

Tue, 09 Dec 2025 16:12:01 +0000

Fourteen days have passed, and it is time to present a fresh collection of readings that could influence developments in the field of artificial intelligence.

This newsletter explores the evolution of agentic AI systems, provides valuable insights into the Chain-of-Thought (CoT) approach, Vibe coding, and discusses the pattern-matching capabilities of LLMs. The newsletter features an insightful interview with Stuart J. Russell, known for his significant contributions to the AI field. Even more exciting is the published paper by Apple researchers titled ‘The Illusion of Thinking…’ and several immediate reactions to the authors’ conclusions, which allow newsletter readers to observe current research challenges and scientific community responses. This provides readers with a vital picture of the state-of-the-art in AI research.

JC-AI Newsletter #10

Wed, 26 Nov 2025 18:39:40 +0000

Fourteen days have passed, and it is time to present a fresh collection of readings that could influence developments in the field of artificial intelligence.

This newsletter focuses on examining how agentic AI systems improve accuracy, tutorials on agentic system architecture, and importnat security challenges arising from increased not only from agentic AI systems adoption. This edition of the AI newsletter includes compelling discussions and interviews about the future of AI and approaches.

Intro to RAG: Foundations of Retrieval Augmented Generation, part 1

Tue, 27 May 2025 14:42:43 +0000

Retrieval Augmented Generation (RAG) may sound complex, but it accurately represents the process of the system. RAG is a method that enhances the capabilities of Large Language Models (LLMs) by integrating them with external knowledge sources.

Each term represents a piece of the puzzle:

Retrieval - data retrieved from some external source outside the LLM (most often a database, but can include files, webpages, etc)
Augmented - “augmenting” (or adding to) an LLM’s training data. This could include recent or private information that it did not have access to during its training period. Most often, this is done by adding the data to the prompt (or input) to the LLM.
Generation - this is where LLMs are exceptional. They generate a response (text, image, video, etc) that is similar to the data being provided in the input. This is generated from probabilities, so cannot guarantee 100% consistency.

Instead of relying solely on the model’s internal training data, RAG retrieves relevant information from databases or document collections to ground its responses in factual and up-to-date content. This approach not only improves the accuracy and reliability of the generated outputs but also allows the system to adapt to specific contexts or domains, making it a powerful tool for many personal and professional applications.

Event-Driven Architecture and Change Data Capture Made Easy

Tue, 18 Feb 2025 16:44:14 +0000

Hello again! In this article (Part 1), we will discuss two common ways to build modern software systems : Event-Driven Architecture (EDA) and Change Data Capture (CDC). They serve different purposes but can work well together in some situations.

we’ll explain EDA and CDC, discuss their use cases, and explain when to use them and when not to. Using plain language and relatable examples, we’ll also explore how they work together.

Foojay Podcast #56: Vectors in Java Code, Databases, and LLMs

Mon, 08 Jul 2024 07:33:28 +0000

In this Foojay podcast, we enter the world of mathematics by discussing Vectors and how they are crucial for AI and machine learning.

As ChatGPT explains: “A Vector is a mathematical structure that holds numerical values. Vectors are fundamental to the field of Artificial Intelligence, as they allow mathematical operations to be performed efficiently and form the basis of many machine learning algorithms.”

OK, but… how are these vectors crucial for the whole Artificial Intelligence evolution?

Writing a Data Orchestrator in Java

Thu, 02 May 2024 19:43:32 +0000

A data orchestrator is a platform to schedule, organize, and monitor data-oriented workflows. A workflow is a set of tasks, most data orchestrators come with built-in tasks for a wide range of technologies and provide support for custom processing via a scripting language. A data orchestrator can have multiple types of triggers to start a workflow.

Most Data orchestrators are written in Python.

Most Data orchestrators mandate that you write Python code.

Managing Data Residency: The Demo

Mon, 29 May 2023 07:33:28 +0000

I explained the concepts and theory behind Data Residency in a previous post .

It’s time to get our hands dirty and implement it in a simple demo.

The sample architecture

In the last section of the previous post, I proposed a sample architecture where location-based routing happened at two different stages:

The API Gateway checks for an existing X-Country header. Depending on its value, it forwards the request to the computed upstream; If no value is found or no value matches, it forwards it to a default upstream.
The application uses Apache Shardingsphre to route again depending on the data. If the value computed by the API Gateway is correct, the flow stays “in its lane”; if not, it’s routed to the correct database, but with a performance penalty as it’s outside its lane.

I simplified some aspects:

Managing Data Residency: Concepts & Theory

Tue, 16 May 2023 09:18:20 +0000

Cloud computing has opened a Pandora’s Box of many original issues compared to sound old on-premise systems. I believe that chief among them is Data Residency, or Data Location.

Data localization or data residency law requires data about a nation’s citizens or residents to be collected, processed, and/or stored inside the country, often before being transferred internationally. Such data is usually transferred only after meeting local privacy or data protection laws, such as giving the user notice of how the information will be used and obtaining their consent.

Ingesting Data from Kafka into the QuestDB time-series database

Tue, 28 Mar 2023 09:38:35 +0000

If you are working on a project with fast or streaming data, chances are Apache Kafka is already part of your pipeline .

But if you want to analyse your data, you will need to ingest from Kafka into some destination.

For time-series data, QuestDB can be an excellent choice.

For those of you who are not yet familiar with QuestDB , it is an Apache 2.0 licensed database designed for high throughput ingestion and fast SQL queries.

Structured Error Messages for HTTP APIs

Tue, 01 Nov 2022 11:01:18 +0000

Ever since I started to work on the Apache APISIX project, I’ve been trying to improve my knowledge and understanding of REST RESTful HTTP APIs.

For this, I’m reading and watching the following sources:

Books. At the moment, I’m finishing API Design Patterns . Expect a review soon.
YouTube. I’d recommend ErikWilde’ channel . While some videos are better than others, they all focus on APIs.
s. Most RFCs are not about APIs, but a friendly person compiled a list of the ones who are .

Today, I’d like to introduce the “Problem Details for HTTP APIs” RFC, aka , RFC 7807 .

Storing 1TB in Virtual Memory on a 128GB Machine

Tue, 27 Sep 2022 09:10:43 +0000

If you use a standard JVM like the Oracle JVM or the OpenJDK, you might find that as the heap size grows the performance of your JVM can drop as GC pause time escalates.

This tends to be a problem around 32 GB of heap, but it often depends on the application at which point the size of your heap becomes a problem.

One way around this is to use an always concurrent collector such as Azul Platform Prime which is designed to both scale to much larger heap sizes and even reduces the GC pause times for smaller heap consistently.

MicroStream - Part 5: Caching, Integrations and Clustering

Wed, 06 Jul 2022 07:11:38 +0000

In this last article of the series, we cover a few other MicroStream features: caching, clustering, and the integrations into other frameworks.

We have now covered three main aspects of MicroStream:

We have seen how you can configure the StorageManager to define where data is stored.
We have discussed the strategy that you should follow to make sure the changes are also reflected in the storage so that we do not lose data.
We described the serialisation framework of MicroStream that stores the objects in a binary format so that it survives a process restart.

In this last article of the series, we cover a few other features of MicroStream: caching, integrations into other products, and we discuss a major upcoming feature.

MicroStream Next-Generation Serialisation Engine

Wed, 29 Jun 2022 08:07:20 +0000

In this fourth part we go deeper into the Serialisation engine that is within MicroStream to store the Object graph in a binary format.

In the previous articles (part 1 , part 2 and part 3 ), we have already mentioned that MicroStream stores Java instances in storage in a binary way with a new, from the ground up created, serialisation framework.

In this article, we go a bit more in detail about the next generation Java serialisation that we have built to achieve the MicroStream Java Object database and how you can use it outside the functionality of storing the root object that makes up your database.

MicroStream: How to store your data?

Wed, 22 Jun 2022 11:10:55 +0000

In the third article of the MicroStream series, we go into the details what you need to do so that data is stored externally to survive the process restart.

In the previous articles in this series, we introduced the main features of the framework and how you can configure the StorageManager .

Java instances in memory are your database but the StorageManager makes sure that the data are persisted so that they are read the next time the runtime is started again.

MicroStream: Configure the StorageManager

Wed, 15 Jun 2022 08:04:11 +0000

In this second article in the series, we cover how to get started configuring the StorageManager of MicroStream!

As we have discussed in the introduction article of this series, MicroStream provides you with a solution to use Java instances in memory as your database.

You can access the data through getters, the Stream API, or any other method provided on the Java Objects.

This means ultrafast, no mapping required, and no external system for your data.

MicroStream: High-Performance Java-Native-Persistence framework

Mon, 06 Jun 2022 08:03:02 +0000

Within every application, data plays a very important role.

In every industry, you have data, such as customer or product information, analysis results, patient data, or account data is a vital part of the business and the applications that support your business processes.

Data must be in memory to be processed by your application, but must be stored outside of it because the application’s memory is temporary.

Information is lost when the process ends unless we have stored it somewhere else in a sustainable way.