Shai Almog on foojay.io - Friends of OpenJDK

The Codename One JavaScript Port Is Now Free and Open Source

Mon, 03 Aug 2026 15:46:30 +0000

The Codename One JavaScript port is now open source and available on every plan, including Free. You can also build the JavaScript target locally without a Codename One account. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Own Your Pixels: Native Fidelity on Your Schedule

Fri, 24 Jul 2026 15:45:24 +0000

An iOS or Android update can change a screen you shipped without you changing a line of code. If your app builds its UI from UIKit, SwiftUI, Compose, or Material widgets, Apple or Google owns those widget implementations. Codename One does something different. It statically links our lightweight component implementation into your native app. The UI you test is the UI your users keep after the next OS update. An update can still break a platform API or permission contract, but it cannot swap our button implementation for a new one. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

How We Beat HotSpot Performance (By Cheating, But Not Like That)

Fri, 17 Jul 2026 17:24:17 +0000

No, we didn’t cheat in the benchmark. At least I hope we didn’t. Every optimization in this story was gated on bit identical output checksums against HotSpot, and the harness refuses to print a ratio when a checksum differs. If anything, this post is about how good HotSpot actually is. We tilted the table in our favor in every way we could, we hand tuned C code, and we still only beat it on some benchmarks. Getting there was a genuine struggle. If you want to understand the nuts and bolts of what your Java code costs, and the tradeoffs each runtime picks, I hope this is a good read. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

More Surfaces, Same Deal: Cars, Sensors, Commerce, Video And Builds

Fri, 10 Jul 2026 18:17:06 +0000

Last week’s release post was about funding open source without the bait and switch. This week’s release tests that idea again, because two of the new features touch paid infrastructure directly: Commerce and versioned builds. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Funding Open Source Without The Bait And Switch: Analytics, Native Maps, TV And More

Sun, 05 Jul 2026 13:41:00 +0000

Years ago I wrote a piece called Open Source Bait and Switch . The short version: a project is released as open source while it has no business model, a community forms around it, and then the bill comes due. The project either rots into abandonware because nobody can afford to maintain it, or it sprouts a home-grown “source available” license that quietly takes back the freedoms it was given on. You have watched this happen. So have I. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Native Linux, Apple Watch, A Game Builder And Crash Protection

Fri, 26 Jun 2026 18:43:22 +0000

This week brings a native Linux desktop port, an Apple Watch and Wear OS port, a visual Game Builder with a high-level gaming API, and a new crash-protection system, with a tutorial following each one over the coming days. There is also a large piece of work that you mostly should not have noticed: we rebuilt the build cloud, and that rebuild caused a few failed builds along the way. More on that below. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

We Will Not Sabotage Your Code

Sat, 20 Jun 2026 15:37:22 +0000

This is a very low bar. I am genuinely saddened that I need to write it down at all, because until last week it seemed too obvious to say out loud. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Last week the news broke of an open source developer doing something I find reprehensible. The short version: fed up with “vibe coders,” he embedded a hidden prompt aimed at AI agents inside his framework, with instructions designed to destroy data on the machines of people who used it.

Native Java Win32, 3D Gaming, Printing and Wallet

Fri, 19 Jun 2026 15:45:42 +0000

This week we’re introducing native Windows support (no JVM!), a 3D graphics API, a gaming API, support for Apple Wallet, printing and more in what is probably our biggest update ever… But that’s not the thing that excites me the most. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

OpenAPI, ORM, SVG and Lottie

Thu, 18 Jun 2026 00:52:45 +0000

This is the third follow-up to Friday’s release post . Saturday’s was about how you iterate; yesterday’s was about new platform APIs in the core; today’s is about a run of pieces that change how you write the structural parts of an app. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Mac Native Builds, Live Protocols, And Open Issues Under 350

Wed, 17 Jun 2026 15:03:06 +0000

Our focus was all over the place this week with work that targeted many different directions: desktop, monetization, communication, media, and more. This fits with our roadmap of one platform that delivers the promise Java never delivered: WORA for Everything Everywhere. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

AI, OAuth, And Other Platform APIs In The Core

Mon, 15 Jun 2026 09:00:00 +0000

This is the second follow-up to Friday’s release post . It covers the platform APIs that moved into the framework core this release. There are two headline pieces (AI / LLM and the modern OAuth / OIDC stack), and two smaller pieces (WiFi / connectivity and share-sheet result callbacks). This continues the direction the previous release set when we moved NFC, biometrics, and cryptography into the framework core. The full background on that earlier set is in NFC, Crypto, Biometrics, And A New Build Cloud . | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

On-Device Debugging And JUnit 5

Fri, 12 Jun 2026 08:49:00 +0000

This is the first follow-up to Friday’s release post and it covers the two changes from this release that affect how you iterate on a Codename One app rather than what the app itself does. On-device debugging that treats Java as Java on a real iPhone or a real Android device, and standard JUnit 5 against the JavaSE simulator. The first is the one we have been wanting for a long time, and is the one that takes the most explaining, so most of the post is about it. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Metal Default, A New Build Cloud, And A New Format

Mon, 08 Jun 2026 09:47:43 +0000

This week’s release post looks different on purpose. The Friday omnibus has been getting longer and longer, and that has been working against us in two ways. SEO ignores 5,000 word pages that cover twelve unrelated topics, so the actual material gets buried instead of indexed against the queries that should find it. And when a single release post covers ten things, it becomes hard to point a colleague at “that one Codename One change from a few weeks ago” without scrolling for ten minutes. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

NFC, Crypto, Biometrics, And A New Build Cloud

Wed, 03 Jun 2026 08:37:24 +0000

Last week was about defaults. This week is about device APIs moving into the framework core, a small simulator change that revolutionizes Bluetooth development, and a preview of the new Build Cloud UI we would love your feedback on. There is a handful of other things in here too — and the Metal default flip I trailed last week is in a different state than I expected, which is worth a word at the end.

Skills, Java 17, And Theme Accents with Codename One

Wed, 27 May 2026 09:32:00 +0000

Last week was about Metal and the Skin Designer. This week the headline items are about what a brand new project looks like when you generate it: the default JDK is Java 17, and every generated project ships with an AGENTS.md authoring skill that lets any modern AI agent work on the project intelligently. There are also some other things worth covering: a runtime accent palette on the new native themes, three Metal follow-ups (one of which introduces a new matrix-correct translate API), the JDK 11+ String API gap closed, and iOS push permission that no longer fires at app launch.

Metal and Skins

Thu, 21 May 2026 08:59:46 +0000

This post has a lot to cover. Before we get to any of it I want to take on the uncomfortable subject first: quality. Two incidents from the past two weeks deserve a public explanation, one was a bug that fits into our normal iteration loop and one was a serious mistake on my part. Both deserve the kind of explanation I would want if I were on the other side of the import. | What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com .

Liquid Glass, Material 3, And A Lot Of Plumbing

Mon, 11 May 2026 09:10:00 +0000

It has been one of those weeks where the diff is bigger than the headline. The headline is short — Codename One now ships modern native themes: an iOS “liquid glass” look and an Android Material 3 look, bundled into the iOS and Android ports, on by default in the Playground, and selectable from a brand new menu in the simulator. The diff behind that headline is several thousand lines across the platform ports, the simulator, the GUI plumbing, and a small army of screenshot tests.

Extending Java APIs - Add Missing Features Without the Hassle

Fri, 07 Mar 2025 12:03:57 +0000

The Java API is vast. That’s great, but sometimes a missing method or capability can be frustrating. With Manifold, developers can solve this problem without having to wait for Java to add a feature in a later version. Even more importantly, Manifold provides many such extensions out of the box, making Java better for developers. In this post, we will discuss the extension capability. It lets us change classes in the Java API in a compatible way without risk. If Java adds these APIs later then we can seamlessly switch to them.

Front End Debugging Part 3: Networking

Thu, 27 Feb 2025 16:34:04 +0000

Network Debugging Powerhouse
Re-Issuing and Modifying Requests
- cURL and Postman
Throttling and Debugging Race Conditions
- Why Throttling Matters:
- How to Use
Managing State with Storage Tools
- Challenges of State Management
Analyzing Request and Response Headers
Debugging in Incognito Mode: Limitations and Best Practices
Connecting the Front-End to the Database
Final Word

Debugging network communication issues is a critical skill for any front-end developer. While tools like Wireshark provide low-level insight into network traffic, modern browsers like Chrome and Firefox offer developer tools with powerful features tailored for web development. In this post we will discuss using browser-based tools to debug network communication issues effectively. This is a far better approach than using Wireshark for the vast majority of simple cases.

Front End Debugging Part 2: Console.log() to the Max

Tue, 03 Dec 2024 16:28:40 +0000

Understanding Front-End Logging vs. Back-End Logging
Leveraging Console Log Levels
Customizing Console Output with CSS
Stack Tracing with console.trace()
Assertions for Design-by-Contract
Printing Tables for Clearer Data Visualization
Copying Objects to the Clipboard
Inspecting with console.dir() and dirxml()
Counting Function Calls
Organizing Logs with Groups
Chrome-Specific Debugging Features
Final Word

In my previous article , I talked about why Console.log() isn’t the most effective debugging tool. In this installment, we will do a bit of an about-face and discuss the ways in which Console.log() is fantastic. Let’s break down some essential concepts and practices that can make your debugging life much easier and more productive.

strace Revisited: Simple is Beautiful

Tue, 26 Nov 2024 14:58:28 +0000

Understanding strace and its Origins
- A Look Back: strace and dtrace
- Originating from Sun Microsystems
Technical Functioning of strace
- The Role of ptrace in strace
- Comparing with DTrace
Practical Usage and Advantages
- Ease of Use and Accessibility
- Favored in Linux Environments
strace in Action: A Closer Look at System Calls
- Basic Usage and Output Analysis
- Interpreting System Calls for Debugging
Advanced Features and Tips
- Filtering System Calls for Efficiency
- Exploring a Variety of System Calls
strace and Java: A Special Case
- strace with the JVM
- Allocations and Threading in Java
Final Word

In the realm of system debugging, particularly on Linux platforms, strace stands out as a powerful and indispensable tool. Its simplicity and efficacy make it the go-to solution for diagnosing and understanding system-level operations, especially when working with servers and containers.

In this article, we’ll delve into the nuances of strace, from its history and technical functioning to practical applications and advanced features. Whether you’re a seasoned developer or just starting out, this exploration will enhance your diagnostic toolkit and provide deeper insights into the workings of Linux systems.

Front End Debugging Part 1: Not just Console Log

Tue, 05 Nov 2024 14:01:45 +0000

Instant Debugging with the debugger Keyword
Triggering Debugging from the Console
DOM Breakpoints: Monitoring DOM Changes
XHR Breakpoints: Uncovering Hidden Network Calls
Simulating Environments for Debugging
Debugging Layout and Style Issues
- Final Word

As a Java developer, most of my focus is on the backend side of debugging. Front-end debugging poses different challenges and has sophisticated tools of its own. Unfortunately, print-based debugging has become the norm in front-end. To be fair, it makes more sense there as the cycles are different and the problem is always a single user problem. But even if you choose to use Console.log, there’s a lot of nuance to pick up there.

The Art of Full Stack Debugging

Fri, 25 Oct 2024 07:24:06 +0000

Full Stack Development, A Shifting Definition
- The Full Stack Debugging Approach
Frontend Debugging: Tools and Techniques
Debugging Across Layers
Conclusion

Full stack development is often likened to an intricate balancing act, where developers are expected to juggle multiple responsibilities across the frontend, backend, database, and beyond. As the definition of full stack development continues to evolve, so too does the approach to debugging.

Full stack debugging is an essential skill for developers, as it involves tracking issues through multiple layers of an application, often navigating domains where one’s knowledge may only be cursory. In this blog post I aim to explore the nuances of full stack debugging, offering practical tips and insights for developers navigating the complex web of modern software development.

Unleashing the Power of Git Bisect

Tue, 08 Oct 2024 08:15:50 +0000

The Essence of Debugging with Git
Setting the Stage for Debugging
Advanced Usage and Tips
Unraveling a Regression Mystery
Final Word

We don’t usually think of Git as a debugging tool. Surprisingly, Git shines not just as a version control system but also as a potent debugging ally when dealing with the tricky matter of regressions.

Debugging Kubernetes - Troubleshooting Guide

Mon, 26 Aug 2024 16:45:45 +0000

Table of Contents

Identifying Configuration Issues
- Common Causes and Solutions
- Detailed Investigation Steps
Dealing with Image Pull Errors
- Troubleshooting Steps
Handling Node Issues
- Preventive Measures
Managing Missing Configuration Keys or Secrets
Utilizing Buildg for Interactive Debugging
Conclusion

As Kubernetes continues to revolutionize the way we manage and deploy applications, understanding its intricacies becomes essential for developers and operations teams alike. If you don’t have a dedicated DevOps team you probably shouldn’t be working with Kubernetes.

Failure is Required: Understanding Fail-Safe & Fail-Fast Strategies

Tue, 30 Jul 2024 08:08:33 +0000

Fail-Fast
Fail-Safe
Choosing Between Fail-Fast and Fail-Safe
Redefining Failure
- Data Corruption
- Don’t Fix the Bug
Debugging Failure
- Avoiding Cascading Failure
Hybrid in Production
Final Word

Failures in software systems are inevitable. How these failures are handled can significantly impact system performance, reliability, and the business’s bottom line.

In this article, I want to discuss the upside of failure. Why you should seek failure, why failure is good and why avoiding failure can reduce the reliability of your application. We will start with the discussion of fail-fast vs. fail-safe, this will take us to the second discussion about failures in general.

Mastering Serverless Debugging

Mon, 29 Jul 2024 10:14:26 +0000

Introduction to Serverless Computing
Challenges of Serverless Debugging
Effective Strategies for Serverless Debugging
Debugging a Lambda Application Locally with AWS SAM
Final Word

Serverless computing has emerged as a transformative approach to deploying and managing applications. The theory is that by abstracting away the underlying infrastructure, developers can focus solely on writing code.

Wireshark & tcpdump: A Debugging Power Couple

Mon, 08 Jul 2024 14:51:05 +0000

Introduction to Wireshark
- Browser Network Monitors
Installation and Getting Started
Navigating Through Noise with Filters
Deep Dive into Data Analysis
Beyond Basic Usage
- The Basics of HTTPS Encryption
- Methods for Decrypting HTTPS in Wireshark
  - Pre-Master Secret Key Logging
  - Using a Proxy
Integrating tcpdump with Wireshark for Enhanced Network Analysis
The Role of tcpdump in Network Troubleshooting
Final Word

Wireshark, the free open-source packet sniffer and network protocol analyzer, has cemented itself as an indispensable tool in network troubleshooting, analysis, and security (on both sides).

Why is Kubernetes Debugging so Problematic?

Thu, 20 Jun 2024 06:37:37 +0000

The Immutable Nature of Containers
The Limitations of kubectl exec
Avoiding Direct Modifications
Enter Ephemeral Containers
- Using kubectl debug
Practical Application of Ephemeral Containers
Security Considerations
Interlude: The Role of Observability
Command Line Debugging
Connecting a Standard IDE for Remote Debugging
Conclusion

Debugging application issues in a Kubernetes cluster can often feel like navigating a labyrinth. Containers are ephemeral by design, intended to be immutable once deployed.

This presents a unique challenge when something goes wrong and we need to dig into the issue. Before diving into the debugging tools and techniques, it’s essential to grasp the core problem: why modifying container instances directly is a bad idea.

Debugging Kubernetes Part 1: An Introduction

Wed, 05 Jun 2024 06:12:40 +0000

Introduction to Kubernetes and Distributed Systems
Enter Kubernetes
- Why Kubernetes Stands Out
Kubernetes For Developers
- Kubernetes Basics In Practice
Final Word

While debugging in an IDE or using simple command line tools is relatively straightforward, the real challenge lies in production debugging. Modern production environments have enabled sophisticated self-healing deployments, yet they have also made troubleshooting more complex. Kubernetes (aka k8s) is probably the most well known orchestration production environment. To effectively teach debugging in Kubernetes, it’s essential to first introduce its fundamental principles.

Software Testing as a Debugging Tool

Wed, 15 May 2024 07:43:43 +0000

The Intersection of Debugging and Testing
The Debug-Fix Cycle
Composing Tests with Debuggers
Test-Driven Development
Final Word

Debugging is not just about identifying errors—it’s about instituting a reliable process for ensuring software health and longevity. In this post we discuss the role of software testing in debugging, including foundational concepts and how they converge to improve software quality.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers t his subject. If you have friends that are learning to code I’d appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book .

Debugging Using JMX Revisited

Thu, 18 Apr 2024 13:02:35 +0000

The Need for Advanced Management Tools in Development
- Introduction to JMX (Java Management Extensions)
  - Understanding MBeans
- Spring and Management Beans
Tooling for JMX Management
- Getting Started with JMXTerm
Leveraging JMX in Debugging and Management
Exposing MBeans in Spring Boot
Final Word

Debugging effectively requires a nuanced approach, similar to using tongs that tightly grip the problem from both sides.

Debugging Streams with Peek

Fri, 05 Apr 2024 08:05:34 +0000

Table of Contents

Understanding Java Streams
- A Simple Stream Example
What is the peek() Method?
- Debugging with peek()
Uncovering Common Bugs with peek()
- Filtering Issues
- Large Data Sets
  - Addressing Side Effects
Limitations and Pitfalls
Final Thoughts

I blogged about Java Stream debugging in the past but I skipped an important method that’s worthy of a post of its own: peek.

DTrace Revisited: Advanced Debugging Techniques

Tue, 13 Feb 2024 07:46:04 +0000

DTrace Overview
Understanding DTrace’s Capabilities
Performance and Compatibility of DTrace
- Cross-Platform Compatibility
- Compatibility Challenges on MacOS
Customizability and Flexibility of DTrace
- Adaptability to Various Scenarios
- Examples of Customizable Probes
Real-World Applications of DTrace
- Final Words

When we think of debugging, we think of breakpoints in IDEs, stepping over, inspecting variables, etc.

However, there are instances where stepping outside the conventional confines of an IDE becomes essential to track and resolve complex issues. This is where tools like DTrace come into play, offering a more nuanced and powerful approach to debugging than traditional methods.

Building gdocweb with Java 21, Spring Boot 3.x and Beyond

Tue, 30 Jan 2024 12:40:00 +0000

Table of Contents

Introducing gdocweb
- Java 21 and Spring Boot 3.x: Innovation and Maturity
GraalVM Native Image for Efficiency
Deployment: VPS and Docker Compose
Front-End: Thymeleaf Over Modern Alternatives
Final Word

Starting a new project is always a mix of excitement and tough decisions, especially when you’re stitching together familiar tools like Google Docs with powerhouses like GitHub Pages.

When Should we Move to Microservices?

Thu, 28 Dec 2023 14:08:28 +0000

Sometime ago I wrote about modular Monoliths and the value of modern Monolithic architecture . One of the more interesting discussions that came out of that post (and video ) is the inverse discussion: when is it right to still pick Microservices?

Like any design choice, the answer is subjective and depends on many things. But there are still general rules of thumb and global metrics we can use.

Before we get into these problems, we need to understand what it means to have a Microservice architecture. Then we can gauge the benefit and price of having such an architecture.

The Theory of Debugging

Mon, 04 Dec 2023 09:18:49 +0000

In the landscape of software development, bugs are an inevitable part of the journey, and debugging, albeit frustrating at times, is an integral part of the process. There’s no escaping this truth, and the sooner we embrace it, the sooner we can master the art of debugging.

In the next few posts in this series, I will explain the little-known “theory” behind debugging. We all know the practice of debugging (to some degree) but there is also a theoretical underpinning that most of us never learned in University (I sure as hell didn’t). Understanding this theory will help you apply a more methodical approach to problem resolution and will improve your understanding of your code.

Everything Bad in Java is Good for You

Tue, 31 Oct 2023 09:28:44 +0000

Everything Bad is Good for You is a pop culture book that points out that some things we assume are bad (like TV) have tremendous benefits to our well-being. I love the premise of disrupting the conventional narrative and was reminded of that constantly when debating some of the more controversial features and problems in Java. It’s a feature, not a bug…

One of my favorite things about Java is its tendency to move slowly and deliberately. It doesn’t give us what we want right away. The Java team understands the requirements and looks at the other implementations, then learns from them.

Not a Single Trace

Fri, 20 Oct 2023 07:06:31 +0000

The Limiting Factor
Strength in Numbers
Example
Magical APIs
Final Word

Your team celebrates a success story where a trace identified a pesky latency issue in your application’s authentication service. A fix was swiftly implemented, and we all celebrated a quick win in the next team meeting. But the celebrations are short-lived. Just days later, user complaints surged about a related payment gateway timeout. It turns out that the fix we made did improve performance at one point but created a situation in which key information was never cached. Other parts in the software react badly to the fix and we need to revert the whole thing.

Debugging Tips and Tricks: A Comprehensive Guide

Mon, 09 Oct 2023 07:12:23 +0000

Rubber Ducking: The Art of Talking it Out
Moving the Goalposts: Redefining the Bug
- The Evolution of a Bug
- The Process of Redefinition
Flipping the Direction: Multiple Angles of Attack
- The Linear Approach to Debugging
Disruptive Environments: Exposing Hidden Bugs
Leveraging Debugging Extensions and Tools
Disconnect and Reconnect: The Power of a Fresh Mindset
Embrace the Challenge: Finding Joy in Debugging
- Strategies to Embrace the Debugging Challenge
Use a Process
Conclusion

Debugging is an integral part of software development.

The Systemic Process of Debugging

Thu, 28 Sep 2023 07:06:14 +0000

The Importance of Issue Tracking
- Avoiding Parallel Work on the Same Bug
- The Value of Issue Over Pull Requests
Communication: Issue Tracker vs. Ephemeral Channels
- Why We Sometimes Avoid the Issue Tracker
- The Role of Daily Meetings
The Role of Testing in Debugging
- Starting with Unit Tests
- Unit Tests vs. Integration Tests
Final Word

Debugging is an integral part of software development. However, as projects grow in size and complexity, the process of debugging requires more structure and collaboration. This process is probably something you already do as this process is deeply ingrained into most teams.

Eliminating Bugs Using the Tong Motion Approach

Fri, 22 Sep 2023 14:53:39 +0000

Understanding the Process of Elimination in Debugging
- The Basics
- Using External Tools
The Power of Unit Tests in Debugging
- Benefits of Mocking Frameworks
The Challenges with Flaky Issues
The Concept of the ‘Tong Motion’
- Applying the Tong Motion to Debugging
An Illustrative Case: Debugging a Server Performance Issue
Wrapping Up

Software debugging can often feel like a never-ending maze. Just when you think you’re on the right track, you hit a dead-end. But, by employing the age-old technique of the process of elimination, and using the analogy of the ‘Tong Motion,’ we can navigate this maze more effectively.

The Evolution of Bugs

Tue, 12 Sep 2023 03:34:48 +0000

Memory Management: The Past and The Present
Concurrency: A Double-Edged Sword
The Pervasiveness of State Bugs
Exceptions: The Noisy Neighbor
Faults: Beyond the Surface
Thread Bugs: Unraveling the Knot
Race Conditions: Always a Step Ahead
Performance Pitfalls: Monitor Contention and Resource Starvation
Final Word

Programming, regardless of the era, has been riddled with bugs that vary in nature but often remain consistent in their basic problems. Whether we’re talking about mobile, desktop, server, or different operating systems and languages, bugs have always been a constant challenge. Here’s a dive into the nature of these bugs and how we can tackle them effectively.

It's 2AM Do you Know What Your Code is Doing?

Fri, 01 Sep 2023 06:53:54 +0000

Observability isn’t Pillars
Tracing and OpenTelemetry
Background Developer Observability
Enough Exposition
How Does This Help at 2AM?
Final Word

Once we press the merge button, that code is no longer our responsibility. If it performs sub-optimally or has a bug, it is now the problem of the DevOps team, the SRE, etc. Unfortunately, those teams work with a different toolset. If my code uses up too much RAM, they will increase RAM. When the code runs slower, they will increase CPU. In case the code crashes, they will increase concurrent instances.

Debugging as a Process of Isolating Assumptions

Wed, 23 Aug 2023 08:16:22 +0000

Debugging is an integral part of any software development process.

It’s a systematic hunt for bugs and mistakes that may be hidden in the intricate lines of your code.

Much like a hunter and its prey, it requires a precise method and a set of specific tools.

Let’s delve deeper into the fascinating process of isolating assumptions to effectively debug your code.

Before I proceed with this week’s post if you have friends who are learning to code… I published a new book for Java beginners with no prior knowledge (for learning programming from scratch). Each chapter also has an accompanying video and I think there’s no book quite like it for beginners. I would appreciate spreading the word on this.

Is OpenJDK Just a Drop-In Replacement?

Fri, 18 Aug 2023 07:58:02 +0000

I don’t know anyone who is still using the Oracle JDK. It has been my recommendation for quite a while to just switch to an OpenJDK distribution as they are roughly drop-in replacements for Oracle’s official JDK. I’ve repeated that advice quite frequently but I guess I glossed over a lot of details that might be insignificant for hackers but can become a pretty big deal in an enterprise setting.

Can't Reproduce a Bug?

Mon, 14 Aug 2023 07:09:14 +0000

The phrase “it works on my machine” can be a source of amusement, but it also represents a prevailing attitude in the world of development - an attitude that often forces users to prove bugs before we’re willing to investigate them. But in reality, we need to take responsibility and chase the issue, regardless of where it takes us.

A Two-Pronged Approach to Bug Solving

Solving bugs requires a two-pronged approach. Initially, we want to replicate the environment where the issue is occurring; it could be something specific to the user’s machine. Alternatively, we may need to resort to remote debugging or use logs from the user’s machine, asking them to perform certain actions on our behalf.

Building for Failure: Best Practices for Easy Production Debugging

Fri, 21 Jul 2023 15:29:09 +0000

Before going into the content of this post, check out my new book which launched last week!

Quite a few years ago I was maintaining a database-driven system and ran into a weird production bug.

The column I was reading from had a null value, but this wasn’t allowed in the code and there was no place where that value could have been null.

The database was corrupt in a bad way and we didn’t have anything to go on.

GraphQL, JavaScript, Preprocessor, SQL and more in Manifold

Mon, 17 Jul 2023 12:42:59 +0000

We reached the final installment of our Manifold series but not the end of its remarkable capabilities.

Throughout this series, we have delved into various aspects of Manifold, highlighting its unique features and showcasing how it enhances Java development.

In this article, we will cover some of the remaining features of Manifold, including its support for GraphQL, integration with JavaScript, and the utilization of a preprocessor.

By summarizing these features and reflecting on the knowledge gained throughout the series, I hope to demonstrate the power and versatility of Manifold.

How to Share Your Work With a Video or Podcast

Fri, 14 Jul 2023 09:22:11 +0000

Season 2 of the Foojay Podcast has come to an end. Since September of ‘22, 22 episodes where published here on Foojay.

In this article, we want to share some of the experiences of creating such podcasts, but also more generally, how you can share your work in a video with a small (or bigger) budget…

Audio

There is one fundamental rule for the audio: “Garbage in = Garbage out.”

Yes, you can apply many filters and effects in a later phase, but you can’t make a bad recording sound good. You need a clear and good recording as a starting point!

Understanding Security Vulnerabilities: Preventing Attacks

Thu, 06 Jul 2023 12:49:42 +0000

When I was a teenager, our local telephone company introduced a new service - the premium phone calls (AKA 1-900 numbers).

The fun part was that we discovered a workaround to these charges by dialing the sequential local numbers which these 1-900 numbers would redirect to.

If the “support number” for the 1-900 was 555-555 we would dial every number between 555-455 and 555-655 until we hit the jackpot…

Hours were spent dialing these numbers, leading us to make numerous calls for free. This attack is still prevalent today, and it’s called Insecure Direct Object References (IDOR).

Manifold vs. Lombok: Enhancing Java with Property Support

Mon, 19 Jun 2023 18:04:59 +0000

Today, we’ll explore an intriguing aspect of Manifold: its property support.

In comparison, we’ll also examine Lombok, a popular library that offers similar features.

By analyzing the differences between these two projects, we can gain insight into their core philosophies and advantages.

Let’s dive in and discover how Manifold’s property support stands out.

As always, you can find the code examples and additional resources for this post on my GitHub page .

Operator Overloading in Java

Sat, 10 Jun 2023 12:19:26 +0000

In this article, we’ll delve into the fascinating world of operator overloading in Java.

Although Java doesn’t natively support operator overloading, we’ll discover how Manifold can extend Java with that functionality.

We’ll explore its benefits, limitations, and use cases, particularly in scientific and mathematical code.

We will also explore three powerful features provided by Manifold that enhance the default Java type safety while enabling impressive programming techniques.

We’ll discuss unit expressions, type-safe reflection coding, and fixing methods like equals during compilation.

Java String Templates Today

Fri, 02 Jun 2023 16:28:25 +0000

In our last article , we introduced you to the Manifold project and how it offers a revolutionary set of language extensions for Java, including the ability to parse and process JSON files seamlessly in Java.

Today, we will take a look at another exciting feature of the Manifold project: string templates.

But before we get to that, some of the feedback I got from the previous post was that I was unclear about Manifold. Manifold is a combination of an IDE plugin and plugins to Maven or Gradle.

Revolutionize JSON Parsing in Java with Manifold

Fri, 26 May 2023 17:39:19 +0000

Java developers have often envied JavaScript for its ease of parsing JSON.

Although Java offers more robustness, it tends to involve more work and boilerplate code.

Thanks to the Manifold project, Java now has the potential to outshine JavaScript in parsing and processing JSON files.

Manifold is a revolutionary set of language extensions for Java that completely changes the way we handle JSON (and much more…).

Getting Started with Manifold

The code for this tutorial can be found on my GitHub page . Manifold is relatively young but already vast in its capabilities. You can learn more about the project on their website and Slack channel.

Logging Best Practices Revisited

Mon, 15 May 2023 13:07:59 +0000

As I write this my interview on DevCentral hasn’t started yet so if you subscribe to my blog or follow me on socials you might be able to catch it live. If not the recording should appear right here:

Either way, this isn’t the first time I wrote about or talked about logging and the common pitfalls we see when logging in production or debugging. I covered this extensively in the old blog . I also did a video covering these ideas. But my ideas somewhat evolved around some of the concepts I discussed.

Spring Boot Debugging with Aspect-Oriented Programming (AOP)

Fri, 05 May 2023 08:19:18 +0000

Aspect-Oriented Programming (AOP) is a programming paradigm that aims to increase modularity by allowing the separation of cross-cutting concerns.

In simple terms, it helps you keep your code clean, modular, and easy to maintain.

We can leverage AOP to debug Spring Boot applications more effectively and seamlessly.

In this post, we will explore how to use AOP to debug a Spring Boot application effectively.

AOP Concepts

Before diving into debugging with AOP, it is essential to understand the core concepts. AOP lets us write code that executes before or after a method. It includes the following common terms:

Relearning Java Thread Primitives

Tue, 25 Apr 2023 05:26:16 +0000

I’ve coded in Java since the first beta, even back then threads were at the top of my list of favorite features.

Java was the first language to introduce thread support in the language itself, it was a controversial decision back then.

In the past decade, every language raced to include async/await and even Java had some third-party support for that …

But Java zigged instead of zagging and introduced the far superior virtual threads (project Loom) . This article isn’t about that.

Boldness in Refactoring!

Fri, 14 Apr 2023 08:09:09 +0000

The old engineering adage: “don’t touch it, it works”. Is terrible. Don’t listen to it. It might be OK at a small scale but as time goes by the bit rot spreads through your code and servers polluting everything.

Large swaths of your system become “no-man’s-land”. As you’re developing a new system, you must always “touch it” and make sure we hire engineers who aren’t afraid to do so.

Yes, I get it. I said that sentence frequently in the past. I understand the motivation. Management doesn’t care about the bit rot in our future. They care about the here and now. Why are you wasting time on this feature?

DevOps For Developers: Continuous Integration, GitHub Actions

Tue, 04 Apr 2023 07:35:12 +0000

I first ran into the concept of Continuous Integration (CI) when the Mozilla project launched.

It included a rudimentary build server as part of the process and this was revolutionary at the time.

I was maintaining a C++ project that took 2 hours to build and link.

We rarely went through a clean build which created compounding problems as bad code was committed into the project.

A lot has changed since those old days.

Watch Area and Renderers

Sat, 25 Mar 2023 07:25:03 +0000

This is it. The debugging book is live . I would really appreciate reviews and feedback!

I also finished recording and editing the entire course .

There are 50 total videos which total in 7 hours… I also recorded additional videos for the two other free courses for beginners and for modern Java .

So keep an eye on those!

Renderers

In today’s video we discuss one of my favorite obscure IDE features: renderers. Very few people are aware of them. I explained them in the past but I feel I didn’t properly explain why they are so much better than any other alternative. This time I think I got the explanation right.

DevOps for Developers - Introduction and Version Control

Sun, 12 Mar 2023 09:13:33 +0000

I start some of my talks with a joke: back in my day we didn’t have monitoring or observability. We’d go to the server and give it a kick. Hear the HD spin? It’s working!

We didn’t have DevOps. If we were lucky we had some admins and a technician to solve hardware issues. That’s it.

In a small company, we would do all of that ourselves. Today this is no longer practical. The complexity of deployment and scale is so big, it’s hard to imagine a growing company without an engineer dedicated to the operations.

Java Serialization Filtering: Prevent 0-day Security Vulnerabilities

Fri, 03 Mar 2023 16:04:53 +0000

I’ve been a Java developer long enough to remember the excitement when Sun introduced the concept of serialization in the JVM.

In the world of C, we could just write a struct into a file but this was always problematic. It wasn’t portable and had many issues. But for Java we could just write the class and it “worked”. This was pure magic!

Java was still mostly in use in the client side and when we thought about security, we had different vulnerabilities in mind. The sandbox occupied most of our security discussions.

Is it Time to go Back to the Monolith?

Fri, 03 Mar 2023 15:58:20 +0000

History repeats itself. Everything old is new again and I’ve been around long enough to see ideas discarded, rediscovered and return triumphantly to overtake the fad.

In recent years SQL has made a tremendous comeback from the dead. We love relational databases all over again. I think the Monolith will have its space odyssey moment again. Microservices and serverless are trends pushed by the cloud vendors, designed to sell us more cloud computing resources.

Foojay Podcast #14: Debugging Tools and Skills for Fun and Profit

Mon, 20 Feb 2023 08:29:01 +0000

Let’s talk about debugging and observability.

We work with debugging all the time, but how well do we know this common practice?

Observability, monitoring, and debugging at scale for your production…

Podcast Apps

You can listen and subscribe to the Foojay Podcast on:

Spotify
Apple Podcasts
And most others…

Guests

A lot of rubber ducks
Johannes Bechberger
- JVM and profiler developer
- Worked on the JDWP protocol and profiling
- https://github.com/parttimenerd
- https://twitter.com/parttimen3rd
- https://mastodon.social/@parttimenerd
- AsyncGetStackTrace: A better Stack Trace API for the JVM
- A short primer on Java debugging internals
Marit van Dijk
- Developer Advocate at JetBrains
- https://maritvandijk.com/
- https://twitter.com/MaritvanDijk77
- https://mastodon.social/@maritvandijk
Ties van de Ven
- Software Engineer @ JDriven, Coach @ Jcore
- https://www.tiesvandeven.nl/
- https://twitter.com/ties_ven
- 6 Steps to help you debug your application

Host

Shai Almog
- Author of “Practical Debugging at Scale ”
- https://debugagent.com/
- https://mastodon.social/@debugagent
- https://twitter.com/debugagent

Producer

Frank Delporte
- https://foojay.social/@frankdelporte
- https://twitter.com/frankdelporte

Content

00'00 Intro and music
00'24 About the topic of this podcast
00'58 Introduction of the guests and host
05'14 Debugging with IntelliJ IDEA and discoverability of tools
- YouTube: Debugger playlist
- YouTube: Profiling tools
- YouTube: Profiling live stream
13'27 JDWP protocol
19'43 Exception breakpoints
- Foojay: Exception Breakpoint that Doesn’t Suck and a Real Use Case for Method Breakpoints
20'34 External debugging tools
- https://rubberduckdebugging.com/
26'55 Observability
37'58 What information should you look for while debugging
45'46 Be aware of tunnel vision while debugging
49'33 What to do if you don’t know where to search for the bug
- https://git-scm.com/docs/git-bisect
- Foojay: External Debugging Tools 2: git bisect
57'05 Outro

What are you Missing by Debugging in VS Code?

Fri, 17 Feb 2023 16:20:16 +0000

In the first chapter of my debugging book, I discuss IDE debugging.

In that chapter, I mostly talk about IntelliJ/IDEA.

People often ask me why I didn’t write as much about VS Code…

The reason is that there isn’t much to write about. Its debugger is simpler for better and for worse. It isn’t as powerful as other IDEs. I created the following video that covers the content of this post:

Remote Debugging Dangers and Pitfalls

Fri, 10 Feb 2023 10:14:47 +0000

This is the last part of the debugging series, to learn the rest you’ll need to get the book or the course .

One of the most frequently asked questions I receive is: can we do these things in VS Code?

The answer is unfortunately no. But I elaborate on the debugging capabilities of VS Code in this video . I’ll do a blog post that covers that next week.

Below is the last video in the series:

Memory Debugging - a Deep Level of Insight

Fri, 03 Feb 2023 10:08:16 +0000

When I mention memory debugging the first thing that comes to the minds of many developers is the profiler. That isn’t wrong but it’s still a partial picture. Profilers are amazing at mapping that “big picture” but when you want to understand the domain, they fall short.

Modern debuggers let us gain a level of insight into the application that’s unrivaled. We can inspect and locate a specific object instance with surgical precision.

Debugging Threads and Asynchronous Code

Fri, 03 Feb 2023 10:07:22 +0000

I’m doing a community interview in the Code Ranch drop by and ask a question to win a free book!

This week we’ll discuss one of the harder problems in programming: threading. For many cases threading issues aren’t as difficult to debug. At least not in higher abstractions.

Asynchronous programming is supposed to simplify the threading model but oftentimes it makes a bad situation worse by detaching us from the core context.

Debugging Streams and Collections

Fri, 13 Jan 2023 14:59:59 +0000

I will run a book giveaway promotion on the Code Ranch on January 17th . Be sure to be there and let your friends know. It would be great to answer your questions about debugging. I’m very excited by this and by the feedback I’m getting for the course and new videos.

I also launched a free new Java course for complete beginners . No prior knowledge needed. This is probably not the audience for this course. But if you know someone that might be interested I’ll appreciate a share. I hope people find it useful and learn a bit about Java. I’m working on a cadence of one video per week in this beginner course.

The Massive Hidden Power of Breakpoints

Fri, 13 Jan 2023 14:58:57 +0000

It’s been a big week. I’m currently reviewing the final draft of my upcoming debugging book . This is always a sobering, and exciting moment. A moment in which all of a sudden all the months of work become “real”.

I also recorded 9 videos for youtube bringing up the total videos in the course to 29 (with many more on the way). I’m starting two additional free courses one targeted at absolute beginners . Another one about modern Java programming which I’ll launch really soon. Both will be free on my youtube channel (obligatory “like, subscribe and share”).

Debugging Program Control Flow

Fri, 23 Dec 2022 07:36:35 +0000

As of now I published the first three videos of the course and will publish the fourth tomorrow. I plan to publish two videos per week on YouTube to maximize the impact but here I’ll only blog one lesson per week to avoid oversaturation. I shot about four hours of video content and still haven’t finished the 2nd module out of eight. This course will be very detailed and intensive.

Debug Like a Senior Developer

Fri, 16 Dec 2022 06:34:02 +0000

My book on debugging is already on preorder and I’m super thrilled to announce I’m doing a full online course to go along with it.

The course website isn’t ready yet but I already have the full outline and a lot of recorded material. I will try to drop videos at a rate of two per week in the next few months until the full course will be online.

It’s shaping up to be a very detailed course. I recorded the first module and half of the second module and I’m already close to 3 hours of dense recorded material!

SpringOne TLV World Tour Trip Report

Fri, 09 Dec 2022 22:00:00 +0000

My talk was accepted by SpringOne in San Francisco. I never went to that conference and was really looking forward to it. This year would probably be amazing with the Spring 6.0 and Spring Boot 3.0 releases. So many groundbreaking changes. Unfortunately, my travel budget at Lightrun was cut, and I eventually left. This meant I had to cancel the trip, I would have paid for travel, but San Francisco is both far and prohibitively expensive. I’ll have to take a raincheck.

Why I Don't do TDD

Fri, 02 Dec 2022 08:59:00 +0000

I recently gave a talk about debugging for the London Java Community. During the Q&A part of the talk, someone asked me about my approach to Test Driven Development. In the past I looked at that practice in a more positive light. Writing lots of tests. How can that be bad?

But as time moved on, I see it in a different light. I see it as a very limited tool that has very specific use cases. It doesn’t fit into the type of projects I build and often hinders the fluid processes it’s supposed to promote. But let’s backtrack for a second. I really liked this post that separates the types and problems in TDD. But let’s simplify it a bit, let’s clarify that every PR should have good coverage. This isn’t TDD. It’s just good programming.

You’re Doing It Wrong — Recruiting a DevRel

Fri, 25 Nov 2022 14:42:15 +0000

This article was co-written with Wesley Faulkner .

Wesley Faulkner is a first-generation American, public speaker, and podcaster. He is a founding member of the government transparency group Open Austin and a staunch supporter of racial justice, workplace equity, and neurodiversity. His professional experience spans technology from AMD, Atlassian, Dell, IBM, and MongoDB. Wesley currently works as a head of community at SingleStore, and in addition, co-hosts the developer relations-focused podcast Community Pulse and serves on the board for SXSW.

Internal Security: Hardening Internal Systems

Fri, 18 Nov 2022 15:08:28 +0000

60% of security breaches are internal to the organization and yet when we think about security we usually think about vulnerabilities, exploits, etc. All the while, 60% of hacks are just from a person just logging into the system and taking whatever the hell they want.

I’ve been thinking about this quite a bit recently. I spent a great deal of time researching security-related issues for a chapter of my upcoming book and unfortunately found very little on hardening internal systems.

API Mocking: Essential and Redundant

Fri, 11 Nov 2022 08:13:23 +0000

I love contradictions where both states of truth can work at the same time.

Case in point is this tweet about mocking from the other week:

If you use mocks, you are not testing

My answer was:

[Mocks help assert limited fixed functionality and they do it FAST.

Which is what unit tests are about. Mocks are an essential part of a wider quality strategy. Otherwise you will spend your time in integration test churn.

Observability is Cultural

Fri, 04 Nov 2022 15:23:00 +0000

I’m guilty of applying the word debugging for practically anything.

My kids’ legos won’t fit, let’s debug that.

Observability is one of the few disciplines that actually warrant that moniker, it is debugging. But traditional debugging doesn’t really fit with observability practices. I usually call it “precognitive debugging”. We need to have a rough idea in advance of what our debugging process will look like for effective observability troubleshooting.

Note that this doesn’t apply to developer observervability which is a special case. That’s a more dynamic process that more closely resembles a typical debugging session. This is about more traditional monitoring and observability. Where we need to first instrument the system and add logs, metrics, etc. to cover the information we would need as we will later investigate the issue.

When Breakpoints Don't Break

Mon, 24 Oct 2022 07:24:44 +0000

I discussed tracepoints quite a bit in my blog and videos.

They are wonderful, but I feel the nuance of non-breaking is a bit lost.

The true power of this amazing tool is hidden due to our debugging habits and our preconceived notions about debugging. It’s indeed difficult to make the mental shift required for these tools.

The payoff for that mental shift is tremendous when dealing with “weird” bugs. Especially in large systems and with concurrency related issues.

The Reason Java is Still Popular

Mon, 10 Oct 2022 16:11:32 +0000

A great time to post this, right after the release of Java 19, yes, another “my language is better” post. No, I didn’t want to write it. But sometimes people’s bad projection gets the better of me.

In this case the post started as a comment and I ended up deciding turn it into a post. This was strengthened with this post which mostly complains about Quarkus (somewhat unfairly if I might add).

Great Time at JavaZone 2022

Fri, 23 Sep 2022 07:57:36 +0000

I had a pretty awful travel experience trying to reach Oslo, but it’s all worth it for what is possibly my favorite conference: JavaZone.

I absolutely love this conference. It has everything: Sense of humor, craziness, heavy metal, continuous integration of food and amazing talks with great people.

This years conference has a fantasy theme which fits into the “weirdness” of the conference. Notice that when I say “weird” this is one of the highest complements I can give a conference. I a world of cookie cutter conferences JavaZone is unique in every way. See this opening scene in the morning of the first day…

Open Source Bait and Switch

Fri, 16 Sep 2022 07:44:39 +0000

I was reading this article and wanted to post a comment but I felt this warrants a response article.

First, if you don’t know me I’ve written a ton of Open Source code. A whole platform and then some. I think the general view expressed in that article and a lot of the fluff I see online is over simplistic and dangerous.

We Need to get Paid

Who will pay your salary?

Serverless is the New Timeshare

Fri, 09 Sep 2022 07:53:18 +0000

We have a shared amnesia. When I speak to younger developers about past technologies, I often get blank stares. To be fair, some of that is because I’m a bit “intense” or “weird” but some of that is because. Huh? Really? Did we have that?

Case in point XA transactions and 2PC (Two Phase Commit). We have a young generation that’s completely ignorant of that capability and the fact that this was “a thing”. Did the requirement for transaction management somehow vanish?

Migrating Monoliths to Microservices in Practice

Fri, 02 Sep 2022 14:47:21 +0000

There have been amazing articles on the subjects of migrating from a monolith to a microservice architecture, e.g., this is probably one of the better examples .

The benefits and drawbacks of the architectures should be pretty clear. I want to talk about something else though: the strategy.

We build monoliths since they are easier to get started with. Microservices usually rise out of necessity when our system is already in production.

External Debugging Tools 3: JMXTerm

Fri, 26 Aug 2022 14:52:23 +0000

When tracking a bug we need to take a two pronged approach. Similar to tongs that wrap the buggy module from both sides and squeezes to find the problematic part. Up until now we discussed tools that are very low level. Some can be used to debug system level services. Today we’ll discuss the other side of the stack but still a very advanced management tool. To understand this you need to understand the field we’re in.

Bridge the DevOps/Dev Gap without Sacrificing Reliability

Fri, 19 Aug 2022 09:16:00 +0000

DevOps culture revolutionized our industry. Continuous Delivery and Continuous Integration made six sigma reliability commonplace. 20 years ago we would kick the production servers and listen to the hard drive spin, that was observability. Today’s DevOps teams deploy monitoring tools that provide development teams with deep insight into the production environment.

“O brave new world That has such people in’t!”

— William Shakespeare

Before DevOps practices were commonplace, production used to fail. A lot. We don’t want to go back to the time before DevOps tools were commonplace…

When Disaster Strikes: Production Troubleshooting

Fri, 12 Aug 2022 11:03:08 +0000

Tom Granot and myself have had the privilege of Vlad Mihalcea’s online company for a while now. As a result we decided to do a workshop together talking about a lot of the things we learned in the process. This workshop would be pretty informal ad-hoc, just a bunch of guys chatting and showing off what we can do with tooling.

In celebration of that I thought I’d write about some of the tricks we discussed amongst ourselves in the past to give you a sense of what to expect when joining us for the workshop but also a useful tool in its own right.

External Debugging Tools 1: dtrace and strace

Fri, 05 Aug 2022 13:13:43 +0000

Often when debugging, we need to step outside of the comforting embrace of the IDE to reproduce or track an issue.

In this series, I’d like to go over some tools you might find useful for these cases. I’ll try to limit myself to tools that are 100% debugging tools and not those that are useful for development testing.

E.g., tools like curl or jq are remarkably useful. You can/should use them while debugging. But you probably used them while building and testing the feature. So you would already be familiar with them and should have some sense of what they do. I want to focus on tools that you would mostly reach for when debugging. In that sense, tools like SDKMan etc. also make no sense here.

Understand the Root Cause of Regressions with Git Bisect

Fri, 29 Jul 2022 13:18:07 +0000

In this series, I cover a lot of magical tools and git bisect is probably the best example of such magic.

The hardest part in debugging is knowing the general area of the bug… and bisect literally shines a light on the specific commit that caused it!

Before we begin, let’s make one thing clear: bisect is a tool for debugging regressions. It does nothing for regular bugs. When we have a regression, we typically know that the issue used to work in a specific release, we would typically have a specific revision where the code worked. We would typically know that it doesn’t work in the current version, but which commit along the way causes the failure?

The Cost of Production Blindness

Fri, 22 Jul 2022 08:11:03 +0000

When I speak at conferences, I often fall back to the fact that just a couple of decades ago we’d observe production by kicking the server.

This is obviously no longer practical. We can’t see our production. It’s an amorphous Cloud that we can’t touch or feel. A power that we read about but don’t fully grasp.

In this case, we have physical evidence that the Cloud is there.

A part of this major shift in our industry is a change to our fundamental roles as engineers. DevOps and SRE are roles that didn’t exist back then. Yet today, they’re often essential for major businesses. They brought with them tremendous advancements to the reliability of production, but they also brought with them a cost: distance.

Debugging Java Collections Framework Issues in Production

Fri, 15 Jul 2022 08:49:59 +0000

The Java Collections Framework was a huge leap forward when it was introduced as part of Java 2 (JDK 1.2).

Thanks to the included collection classes we finally moved beyond the limits of Vector and Hashtable to more mature and generic solutions. With the introduction of streams and functional concepts in Java 8 the framework took everything to the next level.

One of the core principles underlying the framework is coding to the interface. As such you would use the List interface or a Collection interface instead of the concrete implementation. This is great engineering but it makes debugging the Java Collections far more challenging.

Debugging Gson, Moshi and Jackson JSON Frameworks in Production

Fri, 08 Jul 2022 07:14:01 +0000

Parsing bugs are the gift that keeps on giving in the age of APIs.

We use a service, it works perfectly in debugging, QA, etc.

Then some user input that made its way to the web request returns a result we just can’t parse.

Unfortunately, there isn’t much we can do at this stage. We need to understand why the failure occurred and how we can workaround it and fix it.

Understand Source Code – Deep into the Codebase, Locally & Production

Fri, 01 Jul 2022 07:17:53 +0000

Say you have a new code base to study or picked up an open source project. You might be a seasoned developer for whom this is another project in a packed resume. Alternatively, you might be a junior engineer for whom this is the first “real” project.

It doesn’t matter!

With completely new source code repositories, we still know nothing…

The seasoned senior might have a leg up in finding some things and recognizing patterns. But none of us can read and truly follow a project with 1M+ lines of code. We look over the docs, which in my experience usually have only passing resemblance to the actual codebase. We segregate and assume about the various modules, but picking them up is hard.

Spring Boot Performance Workshop with Vlad Mihalcea

Fri, 24 Jun 2022 07:43:52 +0000

A couple of weeks ago, we had a great time hosting the workshop you can see below with Vlad Mihalcea. It was loads of fun and we hope to do this again soon!

In this workshop, we focused on Spring Boot performance but most importantly on Hibernate performance, which is a common issue in production environments.

It’s especially hard to track since issues related to data are often hard to perceive when debugging locally. When we have “real world” data at scale, they suddenly balloon and become major issues.

Debugging jsoup Java Code in Production Using Lightrun

Fri, 17 Jun 2022 09:35:06 +0000

Scraping websites built for modern browsers is far more challenging than it was a decade ago.

jsoup is a convenient API that makes scraping websites trivial via DOM traversal, CSS Selectors, JQuery-Like methods and more. But it isn’t without its caveat. Every scraping API is a ticking time bomb.

Real-world HTML is flaky. It changes without notice since it isn’t a documented API. When our Java program fails in scraping, we’re suddenly stuck with a ticking time bomb.

Remote Debugging and Developer Observability

Fri, 10 Jun 2022 10:12:26 +0000

In this post, we’ll go over remote debugging. Debugging production with developer observability tools and debugging asynchronous code which presents its own unique challenges!

This is actually material from four separate videos because I wanted to pool some ideas together into a single blog post and the asynchronous stuff got pulled in. So without further ado, these are the applicable videos… Read on for the analysis and discussion:

🦆 Duckling the 15th:

Debugging the Java Message Service (JMS) API using Lightrun

Fri, 03 Jun 2022 09:26:12 +0000

The Java Message Service API (JMS) was developed by Sun Microsystems in the days of Java EE .

The JMS API provides us with simple messaging abstractions including Message Producer, Message Consumer, etc. Messaging APIs let us place a message on a “queue” and consume messages placed into said queue.

This is immensely useful for high throughput systems – instead of wasting user time by performing a slow operation in real-time, an enterprise application can send a message. This non-blocking approach enables extremely high throughput, while maintaining reliability at scale.

Memory Debugging and Watch Annotations

Fri, 27 May 2022 08:04:34 +0000

Before diving into debugging memory issues and the other amazing running process, memory debugging capabilities (which are amazing)… I want to discuss a point I left open in the last duckling post . Back there we discussed customizing the watch renderer. This is super cool!

But it’s also tedious. Before we continue, if you prefer, I cover most of these subjects in these videos:

🦆 Duckling the 11th:

Did you know you can see all the objects in memory?

Debugging Deadlocks and Race Conditions

Fri, 20 May 2022 08:08:00 +0000

Thread debugging has the reputation of being one of the most arduous tasks for developers. I beg to differ.

Asynchronous debugging is so much worse. It’s supposed to solve the problems of threading and to some degree async helps…

But it doesn’t make debugging simpler. I will get into that in the next post.

In the last two ducklings, I talked about threading issues:

🦆 Duckling the 13th:

Debug race conditions and deadlocks easily.

How To Modernize Legacy Code in Production

Mon, 09 May 2022 15:58:36 +0000

I spent over a decade as a consultant working for dozens of companies in many fields and pursuits. The diversity of each code base is tremendous. This article will try to define general rules for modernizing legacy code that would hopefully apply to all. But it comes from the angle of a Java developer.

When writing this, my primary focus is on updating an old Java 6 era style J2EE code to the more modern Spring Boot/Jakarta EE code. However, I don’t want to go into the code and try to keep this generic. I discuss COBOL and similar legacy systems too. Most of the overarching guidelines should work for migrating any other type of codebase too.

Spring Transaction Debugging in Production with Lightrun

Fri, 06 May 2022 15:00:53 +0000

Spring makes building a reliable application much easier thanks to its declarative transaction management.

It also supports programmatic transaction management, but that’s not as common.

In this article, I want to focus on the declarative transaction management angle, since it seems much harder to debug compared to the programmatic approach. This is partially true. We can’t put a breakpoint on a transactional annotation.

But I’m getting ahead of myself!

What is Spring’s Method Declarative Transaction Management?

When writing a spring method or class, we can use annotations to declare that a method or a bean (class) is transactional. This annotation lets us tune transactional semantics using attributes. This lets us define behavior such as:

Debugging Collections, Streams, and Watch Renderers

Fri, 29 Apr 2022 07:54:01 +0000

In the last two ducklings, I finished the extensive discussion on breakpoints and switched my focus to the watch area. In it, we have several amazing and lesser known tools that let us build insight into our running application. Being able to tell at a glance if something works correctly is crucial for many applications.

This is very important for collections and arrays. We can have thousands or millions of elements within a collection. Debugging this is very difficult without some basic tools.

Exception Breakpoint that Doesn't Suck and Other Use Cases

Fri, 22 Apr 2022 07:37:25 +0000

Two weeks ago, I left this series in a “cliffhanger” of sorts. Well, as much as a programming blog can leave things in the air… The big one amongst them is the premise that exception breakpoints don’t have to suck. If you used them in the past, you would know that grabbing all exceptions is ridiculous. You end up at a breakpoint every second and it doesn’t help.

There’s a solution, and it’s discussed in duckling 6 number 7 also covers a lot of interesting ground for us and another cliffhanger on method breakpoints:

Logging Best Practices: MDC, Ingestion, and Scale

Mon, 11 Apr 2022 12:16:37 +0000

I don’t care about religious wars over “which logger is the best”. They all have their issues. Having said that, the worst logger is probably the one built “in-house”. So yes, they suck, but re-inventing the wheel is probably far worse.

Let’s discuss making these loggers suck less with proper usage guidelines that range from the obvious to subtle. Hopefully, you can use this post as the basis of your company’s standard for logging best practices.

Debugging JAXB Production Issues

Mon, 04 Apr 2022 14:13:10 +0000

Java Architecture for XML Binding (AKA JAXB API) is a popular API for marshalling XML data. It’s a framework for mapping between XML documents and Java POJOs (Plain Old Java Objects, AKA regular Java classes) almost seamlessly. The API is very easy to use and many frameworks leverage it to provide their XML support. JAXB2.0 has gained popularity both in desktop applications (Java SE) and in application server code (Spring Boot, Java EE/Jakarta EE, Microprofile etc.).

Spring Remote Code Execution Vulnerability

Thu, 31 Mar 2022 14:54:49 +0000

I’d like to start by saying that I’m not a security expert. I also won’t link to the exploit. This is a very fresh take on a new vulnerability but there’s already confirmation from Sonatype . The current exploit seems to be limited to Spring on top of Tomcat but it probably can be adapted since the underlying vulnerability seems general enough.

The vulnerability only impacts Java 9 or newer so if you have an older version you should be safe from this specific exploit.

Debugging Java Equals & Hashcode Performance in Production

Tue, 29 Mar 2022 06:50:17 +0000

I wrote a lot about the performance metrics of the equals method and hash code in this article . There are many nuances that can lead to performance problems in those methods. The problem is that some of those things can be well hidden.

To summarize the core problem: the hashcode method is central to the Java Collections API. Specifically, with the performance of hash tables (specifically the Map interface hash table). The same is true with the equals method. If we have anything more complex than a string object or a primitive, the overhead can quickly grow.

Debugging Race Conditions in Production

Wed, 23 Mar 2022 07:45:15 +0000

Race conditions can occur when a multithreaded application accesses a shared resource using over one thread. Unless we have guards in place, the result might depend on which thread “got there first”. This is especially problematic when the state is changed externally.

A race can cause more than just incorrect behavior. It can enable a security vulnerability when the resource in question can be corrupted in the right way. A good example of race condition vulnerabilities is mangling memory. Let’s say we have an admin user name which is restricted and privileged. You can’t change your user name to admin because of validation. But you can change it to anything else…

The Basics of Breakpoints you Might not Know

Thu, 17 Mar 2022 10:41:08 +0000

In episodes 4 and 5 of “140 Second Ducklings”, I got deeper into the more advanced underpinnings of breakpoints. There’s still a lot more to learn to move forward, but even at this stage it’s surprising how many things are relatively unknown in the developer community. And I’m just getting started…

🦆 Duckling the 5th:

Did you know you can disable a breakpoint until you hit another breakpoint?

Do you know what’s a tracepoint?

Developer Observability, KoolKits and Reliability

Wed, 16 Mar 2022 14:01:52 +0000

In preparation for the upcoming Developer Observability Masterclass we’re hosting at Lightrun with Thoughtworks , RedMonk and JFrog , I sat down for a brief interview with Tom Granot - the Director of Developer Relations at Lightrun.

Tom will MC the event as he did for the Developer Productivity Masterclass we ran back in December.

Shai: Tell us a bit about yourself and your background.

Tom: Before joining Lightrun, I worked as an SRE at Actyx. This drove home the importance of observability tools - I got a sense of the potential problems and, to some extent, the blindness that developers face in production.

Detect, Track, Verify Security Issues & 0-Days

Thu, 10 Mar 2022 12:25:09 +0000

Important: You can use Lightrun for free on your servers.

I’m not a security expert. I’d like to think of myself as a security conscious developer, but this is a vast subject with depth and breadth. What I understand is Lightrun and Debugging . In that capacity, I can show some creative ways you can use it as a security tool. A “proper” security expert could take this to the next level.

Java Return Value, IntelliJ Jump to Line

Wed, 02 Mar 2022 15:33:39 +0000

I just published the 3rd episode of the “140 Second Duckling” tutorial series and I’m getting into the rhythm of doing them. I posted the 2nd episode last week and in this post I’ll dig deeper into both.

I tackled a lot of basic stuff about debugging but I picked two big headliners that a surprising amount of developers aren’t familiar with. E.g. How many times did you step over a return statement and cursed?

OSS Debugging Toolkits for Kubernetes

Mon, 28 Feb 2022 12:27:18 +0000

KoolKits (K ubernetes toolkits) are highly-opinionated, language-specific, batteries-included debug container images for Kubernetes. In practice, they’re what you would’ve installed on your production pods if you were stuck during a tough debug session in an unfamiliar shell.

To briefly give some background, note that these container images are intended for use with the new kubectl debug feature , which spins up Ephemeral containers for interactive troubleshooting. A KoolKit will be pulled by kubectl debug, spun up as a container in your pod, and have the ability to access the same process namespace as your original container.

Enter the Cloud Native Dojo: Blackbelt-Level Debugging

Tue, 22 Feb 2022 13:28:42 +0000

Debugging is often viewed as an art form or a craft. This is true for most engineering-related troubleshooting processes (e.g., art of motorcycle maintenance). We’re usually indoctrinated into the basic moves by a senior developer and are then thrown into the proverbial pool. As a result, even senior engineers sometimes have gaps in their debugging skills. There are very few university courses or books on the subject, so it’s really hard to blame them.

What is Debugging in 140 Seconds

Mon, 14 Feb 2022 12:50:58 +0000

I’m launching a new twitter video series that will focus on teaching the concepts of debugging (and other concepts) in small video bites.

This is the first video in the series. I hope to publish a new video every week. It starts with the very basics of debugging and builds up. I think debugging is a powerful tool that even experienced developers don’t leverage enough.

See the video here:

🦆 Duckling the 1st:

Detect and Fix Memory Leaks in Managed Languages

Thu, 10 Feb 2022 08:45:27 +0000

In the previous installment , I talked about the Java garbage collector. In this part, I’ll discuss the most common memory issue: the memory leak. I focus on managed languages, specifically Java, but I will mention some native code tools which are interesting. A memory leak contributes to heap size, which isn’t the most pressing bug in most cases. But when left alone, memory usage can become a problem and, by that point, finding the issue is hard. Unlike a crash dump, where we get a reference to a specific line, a memory leak can remain hidden.

Polyglot Cloud Native Debugging: Beyond Logs/APM

Tue, 18 Jan 2022 07:55:51 +0000

I’ve done quite a few conference talks since I became a developer advocate for Lightrun . One of my most popular talks has a title similar to the title of this article. In this article I’ll try to cover the gist of the talk.

Cloud-Native Applications

There’s practically an alphabet soup of terms covering public cloud, hybrid cloud, etc. There’s also a lot of overlap between the various terms. The nice thing about this topic is that it applies, regardless. You will run into problems with microservices, architecture and monolithic applications or serverless.

Debugging RAM: Java Garbage Collection - Java Heap Deep Dive (Part 1)

Tue, 11 Jan 2022 15:36:04 +0000

There are many excellent articles on Java Garbage Collection, Java Memory usage and generally Java heap. Unfortunately, they are all over the place. They mix architecture, concepts and problem solving as separate pieces. A lot of the material is out of date or doesn’t include pragmatic information for solving problems with the garbage collector. E.g. e.g. pause times, heap space usage etc.

In this post I won’t go into memory leaks. They’re important but this is a different subject I would like to discuss in a post on its own.

Hashcode and Equals: Debugging and Performance

Fri, 07 Jan 2022 13:31:35 +0000

A few weeks ago I ran into this story on reddit that discusses the problem with using the URL class as a key in a Map. This boils down to a remarkably slow implementation of the hashcode() method in java.net.URL which makes this class unusable in such situations.

Unfortunately, this is a part of the Java API specification and is no longer fixable without breaking backwards compatibility.

What we can do is understand the problem with equals and hashcode. How can we avoid such problems in the future?

Fail-Fast Reliable Software Strategy. Debug Failures Effectively

Tue, 14 Dec 2021 15:30:46 +0000

I love cooking and use my Thermomix a lot. If you hadn’t heard about that amazing innovation, it’s a kitchen robot… Well, it’s a magical super cooking machine. When designing the Thermomix, its designers took the approach of fail-safe instead of fail-fast . This is a smart choice in this case, but it has its drawbacks.

E.g., my machine tried to recover from a failure which sent it into an infinite recovery loop. I literally couldn’t pull out the food from the lid that was sealed shut. But normally, it’s one of the most reliable devices I own.

Debugging the Technical Interview: Methods and Cheating

Tue, 07 Dec 2021 09:00:30 +0000

The headline caught my attention right away “I was shocked to catch a candidate cheating in an online interview ”. How do you cheat in an interview? Does Cyrano de Bergerac whisper the answer from outside the camera?

Well, close enough. The candidate was googling answers and copying them into a shared screen, pretending this was his code. Hiding this is lying. That’s never OK. But this really reminded me of a tweet I read ages ago which unfortunately I can’t find… It went something like this:

Developer Productivity Masterclass: Interview With Leonid Blouvshtein

Tue, 30 Nov 2021 09:09:01 +0000

On December 7th, Lightrun will host a free master class with Mykyta Protsenko, Senior Software Engineer at Netflix; Michael Wood, Field CTO, HashiCorp; Ryan Menezez, Software Engineering Manager, Meta; and Leonid Blouvshtein, CTO of Lightrun.

[

](https://www.linkedin.com/events/developerproductivitymasterclas6870717107772907520/ )

I got a chance to grab Leonid and make him sit down for this quick interview…

Q: Tell us a bit about yourself?

I’ve worked in several successful startups while most of my experience is with distributed systems that require high performance. This was after I graduated from the elite 8200 intelligence unit of the IDF.

Book Review: “Effortless Cloud-Native App Development Using Skaffold”

Thu, 25 Nov 2021 12:52:08 +0000

I’m a developer who cares deeply about production. But I’m not a devops and unfortunately I’m pretty bad at that. That’s why when I heard about Skaffold it instantly piqued my interest. Write Kubernetes cloud native apps without well… Writing Kubernetes native apps… Sign me up!

Unfortunately, as we all know. The time to pick up a new technology is when we actually need it and then it’s a rush to get something out. Few of us have time to take off from our busy day to study something and learn something new. That’s why when Ashish Choudhary , the author of the book “Effortless Cloud Native App Development Using Skaffold " asked for reviewers in the foojay.io slack group, I jumped on the opportunity. I can learn something interesting and be productive (this book review).

Customer Miscommunication Leads to Ongoing Nightmare

Tue, 16 Nov 2021 08:28:01 +0000

Production disasters are sometimes those pileups we can see coming from miles away. But the train is moving so fast we can’t possibly stop it. This is one of those stories. In it a combination of bad communication, missing skills, immature technological choices created a pile up.

You can’t debug a specification. You must stay vigilant during the development cycle to find conceptual mistakes and reorient the Titanic.

This is a bit of a different story in the series. When I came up with the concept for production horrors my thoughts were mostly about a single day or a single event that made our production fail. Naturally our mind gravitates towards crashes or issues like the recent Facebook outage. But last time around, I gave the example of problematic caching that led to a billing problem…

Book Review: Why Programs Fail by Andreas Zeller

Tue, 09 Nov 2021 10:18:25 +0000

When I got my new job as developer advocate at Lightrun, one of the first things I asked for was books. Lots of books. Some of them cover my new job and others talk about debugging. I’ve been debugging for decades, but I feel like the theory around debugging is a bit vague. My goal was to bolster my terminology used by academics and peers. A secondary goal is to see how others teach ideas related to debugging.

Production Horrors - Handling Disasters: Public Debrief. Handling failures

Wed, 03 Nov 2021 10:49:21 +0000

Just in time for Halloween failures in production are scarier than most movie monsters. Here’s a personal scary story of a production fail…

Halloween is probably the most appropriate time to start this new series of articles. I talk a lot about theory on Foojay , but when you’re on the front line of a production disaster, it “gets real” very fast. People often think of production disasters as crashes or sites going down like the recent Facebook downtime. While that’s an interesting subject, a lot of these things can go under the radar and hit you like a brick wall.

Debugging Tutorial 1 – Introduction: Conditional Breakpoints & Set Value

Wed, 27 Oct 2021 08:03:53 +0000

In this series, I’ll walk you through the process of debugging applications and finding issues within them. As we debug, we’ll cover the techniques important for most developers. I will cover the following debuggers:

IntelliJ/IDEA – with Java/Kotlin
PyCharm – Python
VSCode – for JavaScript
WebStorm – for JavaScript

These should cover most use cases you’ll run into and some things will seem duplicate/redundant so you can just skip to the applicable section if you feel you “got the gist”.

PSA: The Risks of Remote JDWP Debugging: It's Like a Wide Open Door

Wed, 20 Oct 2021 08:13:29 +0000

It isn’t so much a security risk as it is a wide open door with everything labeled for anyone to take freely and matches to burn it all down…

When I teach or lecture and mention security, I always love the analogy of security as an onion. This is a common cliché yet it’s still a powerful analogy for people who are new to the concepts. The equivalent analogy I would pick for remote debugging is like leaving your front door open with all your valuables piled in easy to spot and well labeled boxes. Calling JDWP insecure doesn’t make any sense. It wasn’t designed to be secure so it isn’t…

Understand Stack Traces and how you can Debug Better With them

Tue, 05 Oct 2021 10:57:28 +0000

Recently a junior developer sent me an obfuscated stack trace and was pretty surprised when I instantly knew the problem and pointed him at the necessary change. To be fair, I had the advantage of being the person who put that bug there in the first place… But still the ability to glean information from a stack trace, even an obfuscated one, is a serious skill.

The stack trace in question was a ClassNotFoundException, that’s typically pretty easy and already tells you everything you need to know. The class isn’t there. Why it isn’t there is really a matter of what we did wrong. In this case, since the project was obfuscated, the bug was that this class wasn’t excluded from obfuscation.

Debug Real World Applications using a Systemic Proven Process

Tue, 28 Sep 2021 08:54:00 +0000

In the Debugger Checklist (Part I) , I introduced some of the high level concepts and reviewed some of the common things you can do. In this part, we’ll get down to the process. Again, it’s important to stress that this is boiled down and concentrated!

I don’t want to discuss issue trackers, unit testing and proper TDD. I think there are many resources covering all of these. The focus of these posts is the debug process itself which often gets neglected as “tools”. There’s a lot of technique that separates the seasoned developer from a junior. Furthermore, there are many small advancements and changes we can’t possibly keep up with. I’d like to cover all of those things (some in future posts).

Debugger Checklist: Bug Solving Process

Tue, 21 Sep 2021 09:37:32 +0000

Win at debugging by following an organized process and leveraging the tools you already have!

I’m a disorganized person by nature. When I follow a process it’s by habit and intuition. But when a debugging problem keeps me up at night and gets me to that state of mind where a career of raising sheep in New Zealand seems like an attractive option… That’s when I need to back off and walk through this process in an orderly fashion. This process never fails. When you walk through it, you can track any problem.

8 Debugging Tips for IntelliJ IDEA Users You Never Knew Existed

Thu, 09 Sep 2021 07:15:54 +0000

As developers, we’re all familiar with debuggers. We use debugging tools on a daily basis – they’re an essential part of programming. But let’s be honest. Usually, we only use the breakpoint option. If we’re feeling frisky, we might use a conditional breakpoint.

But guess what, the IntelliJ IDEA debugger has many powerful and cutting-edge features that are useful for debugging more easily and efficiently. To help, we’ve compiled a list of tips and tricks from our very own developers here at Lightrun . We hope these tips will help you find and resolve bugs faster.