Foojay Podcast #23: Java Profiling and Performance

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] Hello and welcome everyone to the Foojay podcast this one is about to Java profiling and performance welcome to the Foojay podcast all your news about OpenJDK with us today we have some key people in the Java performance engineering space let’s see what we will geek out into today and talk about I figure that it’s a pretty natural flow from our very famous speakers here after they’re introduced themselves will sort of have a natural progression but I also I have some questions to ask so my name is Marcus lagergran and I am a Java champion and I’ve been involved with almost all aspects of about the Java language since well 98 I would say I was a user for 96 to 97 and then we got into JVM land and but I’ve also worked in the you Java language to human Oracle mostly on the Java 8 release which was a very interesting project and I tend to end up in professional situations where my knowledge of the full vertical and all the way down from the language in the app through the basement floor to many that is the JVM where you can’t know anything what’s below my vertical continue is down to Silicon to ones

[1:28] And zeros and while that’s not like a skill you need for high level programming an observation of made throughout the year is it makes you a significantly better high level programmer if you do have this vertical I’m not saying that you’re you should write native calls or think about exactly how to run fast on a particular platform so when you write your code but I’m very convinced and there’s probably science somewhere on this too that if you know what the box does to your high level code you will write better high level code it will it will slick it will be readable it will use the high level language constructs and it won’t have anything that like doesn’t have to be there short and clean basically so when someone comes to me and basically has a background the even if the background is I was hacking demos on the Amiga in the 90s for my own pleasure in my parents basement I know that there is vertical knowledge here and that’s actually something that counts as a plus towards hiring people in even an idle space for me because I know that they can very

[2:44] Quickly pick up fundamental things about performance and scalability on the high level and through their understanding of having been poking at the ones and zeros and at some time and apply it and this sounds like a little bit far-fetched but I’m pretty sure I’m not making this up or falling for my own biases here so our first Speaker today is Chris Newland do you want to introduce yourself Chris yeah I’m Chris Newland I’ve been involved in programming Java I’m pretty much since leaving University in 99 so way back in the Java one days and for the last 19 years I’ve been involved in the market data industry so processing stock market data in real time which is a pretty single threaded straight line speed kind of place to be and as as just some really interesting challenges in terms of dealing with the big feeds and growing growing data volumes and how to keep up in real time and also be able to do interesting things with that very fast stream of data I’m also involved in open source so I’ve created a few tools for my own learning

[4:00] And also to help others one of those is called jit watch which is something that I had the idea for back in 2013 when I first went to heinz’s Jay Creek conference I thought I better bring something along to talk about and that has that’s the tool that lets you look at the decisions that the JVM’s just in time compilation system made when it tried to optimize your program I’ve also made a few other tools like Jacqueline which is a tool for analyzing JVM command lines to help you get out of command line from Hell situations and that’s based on another project called VM options Explorer which is a an interactive dictionary of all the thousands of different tuning parameters that you can give to the JVM you should also mention your extremely important book and that you’ve written in the space I think the last Edition came out 2018 with them Evans and a few others that’s right yes I was one of the authors on the O’Reilly title optimizing Java with Ben Evans and James cough this book is really the first of its kind where you go all the way down to like internal nuts and bolts

[5:18] Or the Java performance and it explains the inner workings of modern Java very well Marcus hurt and I who’s this lady today by the way we did like a book on VMS in general the j-rocket VM in particular in 2010 and people are really been wanting something with VM internals like on a more General path and I would say this is an indispensable book to own if if you’re in the Java performance space I would recommend you to buy it it’s called it’s called optimizing Java right that’s right yeah and it covers all aspects of the runtime which is co-generation memory management and actually goes into modern PCS so I would say even with like Java 20 out and the changes it’s introduced this is a very relevant and applicable book and I actually own this book on dead trees a physical paper copy because I mean that’s a rare honor these days that I bestow only on books that I think are very important for what I do professionally and what we all do professionally so this is definitely one of them thanks Chris and with us we also have Marcus hurt who works now and in Engineering Management at datadog still right is

[6:36] That correct Marcus yeah that’s correct so I am the director of engineering for a few different things at data dog these days so I’m running the continuous profiling product I’m also running the dynamic instrumentation product which is not something we will be talking about today but it’s also very interesting and the client libraries so the part of the software that is running inside of the customer processes for enablement of tracing for example and I’m also still the Project Lead for OpenJDK Mission Control and a Java Champion excellent yeah and you’ve been like you and I bring my story at our job around team career together in 99 so then I’ve had the pleasure to working with Marcus for a lot of years before we dispersed yeah up until four years ago I was still part of the Java platform group and then I got an offer I couldn’t refuse from data we will go back to Mission Control and what you did there and what Oracle did in a bit I suspect but first of all well let’s let’s do the final introduction here and with another true industry

[7:57] Giant here in performance in Java space I’ve got that so come on well I’ve grown a lot you know in other directions the than upwards yeah kind of hard to get rid of so anyway from Crete with us we have Heinz so please introduce yourself as well yes I’m Heinz cabotz I live on the island of Crete as Marcus mentioned my first contact with Java was 1996 sitting in the lab working on my PhD and this fellow student came running and it was like all the excited bouncing up and down you got to see this you got to see this and it’s like point me to this website and it was basically the I think it was like the Sorting applet if you guys remember that it’s like doing the bubble sort and the quick sort and you can see that performance difference between all the different algorithms and I said well you know so what and I basically ignored job for quite a while as long as I could because I was the C plus plus program and everything you know besides you know going directly to the hardware was a waste of time but then I started working at a company and we were doing some

[9:12] Telecom stuff and they there was a new project that came up and they wanted to use Java so but initially it was it’s excratingly slow and don’t forget we had like machines which I don’t know I can’t remember exactly the speed was like 120 megahertz or something like that with you know a really fully spec machine was like 64 megabytes of ram so Java was also very unoptimized in those days so it was really really slow but then what was really cool about Java is that over the years people like Marcus and were making it faster without us doing anything towards it basically we just ran our code and it was twice as fast so that was really really interesting and what we discovered is that simple code idioms you know plain coding often resulted in more optimized code and also sometimes we would would do some optimizations that made sense 20 years ago but nowadays they don’t make much sense anymore so I’ve been involved in the job performance space for a while since 97 I would say because in the beginning every single job of programmer was a it was a performance tuning in a specialist otherwise he

[10:23] Couldn’t use it was too slow I also write a newsletter and I run this little conference called Drake read and I think we’re all going to be there is that right we’re going to be after I created this year I think so we’ll have some fun so thanks for inviting me I I’m really am very very honored to be here with these amazing Engineers yeah well thanks for coming here Heinz and I’m sure all of us our three speakers here have some very interesting topics that I would like to discuss stirring out here the reason you mentioned that the Java was slow and that was the reason we got into Java like for the first part because we realized that our customers and we were working consulting jobs basically and we sort of specialist in Java we saw it coming on taking over more and more because people had these very obvious like development productivity increases in Java compared to C plus plus because of memory management and buffer over on checks oh definitely there you go because now that’s the thing I made it takes on of course didn’t take long before Java apps are getting large and it was very

[11:34] Generic because Java can be made to express a lot of stuff even with the Java one in version one apis you had Reflections and stuff like that so application servers is something that really found new life when Java showed up as like a potential development platform and running an application server like weblogic on like the classic VM that didn’t even have a jit and did have a conservative GC at that at the time it was obviously not something that was future proof so sun was quite early talking about Hot Spot being the next kind of adaptive runtime that would just stuff just in time based on profiling data and it could basically the entire runtime here could basically be optimized from the ritual data you actually have when you’re a runtime another static compiler and we realized that this was the correct vision and then I think we were Java one two years in a row 97 98 something very early ones and realized that sen has exactly Slide by Slide the same Hospital presentation as it did last year okay how hard can this be we had to do it ourselves so that was the guy who hubris

[12:52] That made us go into like this obsessive Java performance space and it’s it’s a long journey but I mean it’s still valid the observations that were made back then that like the potential for optimizing in a runtime compared to the static environment some things are worse some things are better and it’s pretty obvious that you have to guess like make assumptions in the JVM of that this code like will never be more complex than this and in a dynamic run time you don’t really have a closed World assumption since it has a lot of problems there was also this space that like DVM is a very competent runtime it has buffer overruns it has performance yeah it has a platform independent almost baiko language so throughout the years there’s been a lot of efforts trying to deploy non-java languages and runtime as well with varied success it’s interesting to see which shape the history of performance engineering here and I would make the point that they’re really hadn’t been an environment like Java before for implementing this kind of stuff if you look at the academics behind it it’s basically it goes back to

[14:06] Small Talk concept of theoretically can do a lot of good things in a run sampling runtime environment and not that you really can do anywhere else and that really intrigued me and was what got me into this in the first place I think it might be a good idea we all have discussed some some topics that you want to drill downies some facets that apply to you performance and scalability and what you do it could be anything from like you get into a huge customer system and try to help out in a consultant basis or courses that a specific need many people have has sort of produced the reason that you hold the scores in our hands is very good at this and keeping on top of real world needs and educating people about performance and especially Java performance we have Marcus who is as like a tool Smith originally and after the initial JVM stuff because as we know we get a lot of profiling data for free in a runtime environment without hooking in some kind of like very slow extra profiling layers in the best of all possible worlds and how we also these days as of job of 40 or

[15:25] Something how we can doc do your own stream flight recorder events for custom desk which to me was very huge in a couple of software architecture jobs they did in fintech after after it showed up and Chris of course has really raised awareness of how legit especially works and also this optimization book I think it’s like almost a Lifetime Achievement Award to see something on that detail level for the Java platform because a lot of people they don’t look below the runtime layer and there we are there’s 150 apps like two last authentic jobs probably running in some microservice cluster and this Java command lines every app has a lot of java command lines and they’re not really appropriate for any of them and there’s all these cargo culted XX options and God knows what and Chris has probably done more than anyone to try to untangle this and make it sort of clear to the world how these things work so you want to start Marcus because you’re you’re first on the notes here let’s talk a little bit about profiling props props yeah I can talk a little bit about profiling

[16:46] So I had some really interesting discussions over the past weeks with people on my teams and I noticed that people had very different ideas even within the profiling team about what profiling really is which I thought would be very very funny and if you go to Google and just look up the word profiling you’re going to see that you know the definition that comes back is going to be the recording and Analysis of a person’s psychological and behavior characteristics so as to assess and predict their capabilities in a certain sphere or to assist in identifying categories of people which sounds a bit Shady but I think for the software profiling part it’s pretty analogous and you know I would simply state it as the recording and Analysis of a program’s runtime Behavior and we’re simply trying to understand how the program behaves who are trying to record data and why it is behaving that way so we’re trying to analyze that data so that we can improve some aspect of the program for something that costs us and it doesn’t need to be money it’s it is often money but it doesn’t have to be

[17:56] Money any kind of costly resources it can be CPU it can be Memory it can be locking Primitives it can be other potentially latency introducing constructs like thread holding operations and so on so my definition of profiling is much wider than you would typically see with people that think about software profiling most most people would would think of yeah we we’re taking stack traces and we’re taking stack traces at different points in time where we think it matters products and profilers and tools for like almost any programming language right you do C plus plus and things like that as well V team yeah or others absolutely yes plus you could argue that is getting more and more like a run time the more the news equals plus standards come out but in order to limit this discussion to the Java space I think which is like the mission statement let’s let’s talk about when we say applications and stuff let’s assume we have things running on the gdm with those characteristics and I mean let’s let’s more than enough to geek out for a larger than we

[19:10] Have very much but I still think it’s important because but people typically think of profilers is sampling CPU profilers you’re typically you know your singular thread to dump its stack at an interval when you know a certain CPU quanta has been used and then you have some good idea where you’re spending the CPU time but that’s not the only profiling that we have in Java right we have a really built-in very powerful capabilities to do other kinds of profiling as well so we have a really nice neat new allocation profiler that was introduced I think JDK 15 which datadog proposed we have other profiling capabilities there’s also a built-in execution Tracer so what go people would call execution tracing at least so j4 the kind of built-in profiling capability for the JVM is an event recorder so when certain things that we care about typically happen we can get an event now getting all of those events is super expensive so we noticed some of these limitations the data though because we see customers that are using that have runtime profiles that vary quite a bit from you know almost single threaded

[20:39] Applications to hyper shredded applications with thousands of threads running concurrently and some of the limitations became very apparent to us when we sort of had customers running 96 core environmental machines with you know allocating allocating like nuts on every core the built-in allocation profiler kind of broke down and we had to do something but I think it’s important to note that JFR the built-in profiling capability actually contains both what people would consider to be a sort of standard sampling profiler though we can geek out on that a bit as well because it’s it’s not really a CPU profiler so we can talk more about that later if you want to but anyways but it also contains sort of an execution Tracer so whenever we hold the threads for example we can get an event and we typically don’t want all events because it becomes much too expensive and produce too much data so we can have outliers we can decide how long we want the events to last for us to record it we come a long way today from people actually instrumenting and regenerating byte code causing highs and bugs causing

[21:49] IRAs whatsoever and you’ve been very close to that development ever from the start so yeah getting the data cheaply available but it’s both something that the runtime has there it’s just a matter of bookkeeping and providing it but also looking at what technology has brought us to do things like virtualization and operating system development the last 20 years is also I mean we’re an amazing place to deal with profiling for our performance engineering today yeah and the cool part is sure we wanted to support performance engineering for many different reasons we want to support performance engineering for reducing the cost of running a program to make the software run faster better throughput we want to make it from smoother so less latent sales variousness stop the world interruptions less variants we have mechanisms to produce good data because I mean a JVM is traditionally at least on the sun side a hot spot well all jvms that I know are safe Point based and safe points are not totally representable as as a sampling points and they were still the only samplings place where you

[23:05] Could safely collect data from exactly and that is changing too which is funny that’s more of the on the execution side of things and it gets worse worse because if you’re trying to implement a basic profiler on the Java level you will never see safe points it will be as if God himself stopped the world and the only noticeable difference will be that when it all wakes up again the wall clock will have jumped and that’s the only synchronous profiling is one of the breakthroughs here when it comes to actually trusting in the data that you get because I mean this is a fairly recent development right well yeah GeForce been around for a very long time and we’re already always had events for safe points so it was actually a funny thing that happened so we kind of have a very opportunistic profiler at data dog it uses a lot of different techniques and Technologies one of the techniques is JFR because some of the data that we get from there is things like save point events but because the execution sample event in JFR has some limitations and we also have our own CPU profiler that

[24:18] Emits data in JFR format but isn’t using j4 you need both but another I think interesting development in profiling in general is that you don’t just use it for performance engineering anymore you also use it to Aid in Diagnostics and this is really cool right if you continuously profile you can find the unknown unknowns using profiling yeah and Trust the data that is my point I mean these days I mean I’m an assertive industry damaged from like save point-based profiling in environments where I had nothing else and I know that the world has been massively led a straight because they think they see reliable data where it’s really quite biased due to execution Behavior so this is what I mean by this ties into what I said about the vertical and the beginning you sort of have to understand the difference yeah we’ve come a long way in profiling for to avoid safe pointing profilers for example that will only show you or trick you into believing that the sample point was actually at the safe point where it might not exactly happen the other thing is the resolution of data and the

[25:37] Abundance of free or near free data this is also something that could not have been taken for granted during traditional profiling and you are one of the people who have spent significant part of your work life and making sure that actually we I actually can do this at all these days yeah but all profilers will lie to you and you have to decide in which way you want to be lied to well I mean we’re trying to do our best of course but at some point so the Heisenberg observer effect is always present right you can’t observe a system without affecting it that is simply not possible so what you’re trying to go after is something that has low enough overhead that it won’t materially affect the shape of your profiling data because then you will simply be led up the bar you will bark up the wrong tree right so you definitely want don’t want that to happen but you are going to need to do to make trade-offs and you probably still have to understand where your data comes from like how is it mined and Chris and Heinz might have a few comments here on what you discussed do you think yeah is

[26:56] There anything you like to add or elaborate I don’t know what Marcus talk about profiling and data oh yeah you have to always take into consideration that you’re being lied to by all the profilers it’s it’s really fun because you can take one problem and throw you know five profiles that you almost get six different answers I think for profiling it’s always useful to First decide what you’re actually looking for before you start looking for it because if you just turn on all the switches and say tell me everything about the system you’d normally don’t don’t do very well with it so first having sort of a hypothesis of what you’re looking for what the causes of your particular issue so that you can hook into that and say I want to know only about for example thread contention I want to only know about object allocation within a certain area it makes it much easier than just saying well just tell me everything you also have to take into account in Java that the runtime and I mean normally in the static compile environment of course the OS is doing

[28:01] Stuff there might be issues with like libsy on your platform that might do a lot of ring transitions to the kernel and then that turns out to be a problem yeah I was actually on the floor when we had to do an emergency effects to the spectrum of his prediction bug where the Linux kernels only with 20 slower for our application yeah some stuff like that style where things are coming from in Java garbage collection can be a significant part of your run time and other factors and you really really want to run your program as much as you can and that’s basically the underlying goal all Java performance improvements I guess yeah so Heinz could you tell us a little bit what you have seen and what motivates like your customer interest in the various courses you held what do you look for when you get an idea okay this is definitely it needs to be an educational segment at the bucket produce and end up let people buy all right to hear you talk about yeah that’s that’s an interesting topic because it’s you know you have Consulting and training and there’s a

[29:12] Very different products all together I sort of slipped into the training business by accident my professor started a training company and he was looking for some sucker to go and teach Java it was Bruce Eccles course at the time and then I’ve been teaching Java for over 20 years and what I’ve noticed is they’re two types of people there are those that want to learn a lot of things really quickly and they’re those that go very very deep into particular topic so you get people like Marcus margarine who goes very very deep and then others who just touch the service but they’ll learn a whole bunch of things really fast and almost all of my customers are those that learn quickly they don’t often don’t even have great books like what Chris Newton wrote they didn’t have those books they just sit there they listen but they absorb very fast and then they’re going to apply it to the real world and you need both you need those people who very deep very big very deep and those that are on the surface but they can cover a very big ground very quickly you know a little bit about like the

[30:15] Software you’re you’re maintaining or developing you can still get very far with very privilege yeah exactly exactly you can get really far with it if you decide a bit so I I’m I’m a bit like Marcus I don’t go as deep but also want to go very deep into topics yeah the thing is it’s it’s like a whole ecosystem because the cons the Consulting I’ll do if a company has his issues I want to speed up this system or they want to solve particular performance bugs that type of discoveries well then as Marcus say push me into direction to say well this actually is something which I’ve seen quite a few times now we need to put that into a course but most of the time what customers will want is for their Engineers to be educated so that they don’t fall into the same trap over and over again sometimes commercial deliveries take proceedings and then you sort of have to have a little bit different approach but you can do a whole probably good there as well if you shoot them show them some quick and dirty tricks and if you just like plug in a few things that you didn’t have well there used to be a lot

[31:22] Of tricks in the past but not as a chicks are somewhat limited because Jarvis becomes optimized it was really in the past they had all the slow hanging fruit and you guys put the switch on in your 25 faster but that’s that was in the past not as Java has improved so much that you can’t optimize as much necessarily anymore what I find interesting is if you take something like the garbage who’s mentioned that they’re gone you can take different garbage created logs for different garbage detectors take G1 and zgc and whatever else parallel GC The throughput Collector zero Tracy and you look at the GC logs and you look at the overhead percentage overhead of each GC and what you’ll discover is that you might find for example that G1 has is more efficient has less overhead supposedly than the throughput collector but your response rate is worse for example so you would always have to compare apples with Apple so like if you’re tuning something if you’re tuning your system and using G1 you need to compare the G1 log to the G1 log after you’ve optimized it you

[32:33] Can’t compare G1 to zgc at that rate you can only compare response rates not not the GC logs and it’s again one of those things where you get kind of light a little bit but not exactly because it’s just very different products and you can’t compare overhead and percentage of time and GC between multiple ten different juices because all work very differently exactly Apples to Apples and sort of standing withdrawal and did not Chris do you have anything to say about this because I think your interest in this story is like you saw complexity and you wanted to entangle and understand it for your own sake and then suddenly what you were working on in your spare time turned into like massive value ads for a lot of people yes yeah I mean my interest in performance came out of necessity because I was working on a trading system at the time which wasn’t performing and that led me to Really sort of ask what is the JVM how does it execute my program another thought this morning before the podcast doing profiling is really distilling the essence to get

[33:40] The behavior of a program you’re trying to extract the signal from the noise it’s it’s a lossy measurement that’s going to help you get an idea of where to look so once you’ve built that profile you’ve distilled the essence of the behavior of your program then you can start thinking well what am I optimizing for am I optimizing for latency and hypertonizing for memory usage and optimizing for throughputs things like that so it’s that that’s that’s sort of my where I start with profile now is I think okay let me get let me get the overall layer of the land and I saw a I saw a nice quote on Twitter a couple of years ago they said they use JFR and jit watch as a one-two punch they use these JFR to gather that profile to understand what the big picture is and once they’ve identified some interesting features in their runtime behavior that’s when they can bring out the second line tools as your first line tools I mean the last eight years they spend on the Java customer side and financial technology trying to apply my skills there in

[34:47] Software architecture’s general which was a lot of fun but frequently in that space people are very horizontal they have no they have no fear of plugging in black boxers let’s say Kafka and whatever is in every huge microservice based app these days and integration and did like Java really works and this way because and the world is incredibly horizontal with products yeah our products has sold most of your problems you have you can concentrate in your business logic this is the revolution that we saw at the end of the 90s where we ran into massive performance issues right and it’s pretty exciting that you can build software with so much complexity but still keep track of it so profiling is also of course mapping how these different boxes talk to each other but it’s slightly outside the scope of this today but what you do know is even like in 2020s on the Java customer side is that well it ends with the JVM that’s the last part here in the stalker file and this is the last thing Orchestra you’re in kubernetes and here is a list of flags that’s long as my arm

[35:58] And you probably have some some to say about that but find that people need to again I don’t think they need to understand everything about the JVM but just like cargo quilting stuff from stack overflow when you set up your critical fintech applications of today may not be the best idea I think what’s really important for people to understand and it’s not obvious to every developer is that the JVM contains a system called ergonomics which tries to look at the environmental execute in and pick some sensible defaults because you’ve got to remember that Java is available now for literally everything from a tiny little Raspberry Pi zero very limited device right up to huge huge servers with hundreds of course and you can give it the same piece of bytecode and the JVM will look at its execution environment try and work out the capabilities of that environment how many CPUs what kind of ram it’s got and try and make some decisions on what garbage collector do I think this program will benefit from what jit defaults should I use so yeah the and because we’re moving into a

[37:04] Different world to the one that I started out in where you’d run Java on a server you could touch and now there’s so many Java programs possibly even the majority are running in containers in the cloud and actually the only last night I saw a very interesting email on the hot spot div mailing list the Java Engineering Group at Microsoft they are discussing introducing a new JEP called ergonomics profiles which is looking at a way to enhance ergonomics possibly rather than the current defaults which are the maximum Heap is the one quarter of the physical memory available determined when the JVM started up with these new enhanced ergonomics perhaps if you’re running in containers and the container was sized and because that’s what it thought the JVM needed then that could make more be a little bit greedier perhaps not make the assumption that you’re running on a shared system but actually make try and fit the JVM to its container more suitably which is a problem I’ve encountered in the real world when we started containerizing some of

[38:10] Our jvms it’s how do you match the container specifications to the ergonomic system of the JVM and I’m and the JVM gives you about 2000 switches that you can possibly poke to make things work the way you want but perhaps we are getting to the stage where we do need to improve the capabilities of the ergonomic system I think that’s actually a very good point because this is something that has taken significant effort for me to fix when I’m taking over various fairly large platforms or some kind of technical leadership role and I don’t think people are aware and at all how much these things can matter here one thing that that’s been interesting is like Dynamic compute power now we have I don’t need to wait in line for a massive amount of large servers like in the 90s that’s our staging environment when I need to do performance runs I can actually go out and say kubernetes or dynamic compute pool whatever give me free machines they’re pretty damn huge that will keep for at 20 minutes yeah running my performance checking tests and then I’ll

[39:23] Disable and put them back in the compute pool so there’s also really interesting these days even why we don’t have the tools mechanisms and stuff for performance engineering we still have the possibility to do that kind of thing to fit them in and I think that is an area where people generally haven’t realized how incredibly important tool is especially for sizing but you said it’s not economic reasons so what what’s interesting right now like what do you see it could either be like adoption something healthy adoption of traditional knowledge like really hasn’t been very simple to fit into software or it could be something new some Alpha technology that you feel with the incredibly important in the next few years for Java and JBL apps Could you yeah could you mention a couple of things that you find is really exciting technology right now or yeah traditional wisdom finally having the mechanisms to be properly applied like what is spreading what is trending and what should be trending what should we try to raise awareness about that like this is easy these days guys try this I

[40:40] Think as the systems get more complex you can’t really ignore the distributed component of this anymore stop running Java Aid in the basement but yeah yeah I mean really these days it’s it’s it’s kind of you gotta have the context of the distributed operations that you’re trying to do and that means that context becomes more and more important for a profiler to be useful you can’t really get as much mileage out of it if you just look at it from the unknowns unknowns perspective the global profiling point of view so I think distributed tracing and the context of the distributed operation is starting to become key which means that I think GFR especially needs to evolve to deal with context and to pick up context we really need to be able to inject context and we talked about this back in Stockholm 4J rockets in the Ws times we were discussing whether or not we should have thread coloring I think it was a mistake not to do it back then and we should have done it but you know we had other means to get around it for example emitting contextual events everywhere

[42:02] Now the problem right now is that you have Frameworks like RX Java reactive programming where context is being switched way too often for us to emit contextual events and to line things up that way it’s it’s becoming too expensive so picking up my app context I think is one of these things that profiling apis and Frameworks we need to deal with and for JFR to be relevant for the future that needs to be part of it also deciding whether or not to emit the event based on if there is context is also going to be necessary I think for all the events in JFR what would straight coloring have been basically I don’t remember the width but having a means to put put a number associated with a thread that could be picked up by JFR by by all our events so that we could could have some way of picking up that context and now you’re going to ask why don’t just use a thread local and yeah you know if that would be integrated I would not have asked that no okay very good thank you so much I would point out that that’s something I see especially in fintech where people

[43:23] Like want to use Java is that they have to try to trick the gdm to do like we have gc3 periods that we can handle the garbage collect during warm up and that I mean that’s a very deep and ingrained pattern you have that object pool that just Ages which is really not anything the Java is built for and in general is pretty damn hard in memory management to keep stuff alive anyway but then like recently and this is probably an outlier to the zgc project the JVM as of maybe 17 has gotten so incredibly parallel I mean before when you stop the world that you do complex easy operations you had to do stack walking yeah you had a lot of things that couldn’t see each other you could next keep code at all in the beginning at the same time as you stop the world not now I mean there have been absolute breakthroughs and how much gvm can keep on executing in every part of the runtime not just like the way the GC handles it when you start a safe point which actually makes a lot of apps I’ve seen well if you run leads when you have a 17 and you can garbage collect your heart out you don’t really have a

[44:37] Latency problem anyway you still have very deterministic pause times anyway you don’t need you like resort to this the this hacky object cooling or whatever it is due to get performance these days and that is not I mean an easy thing to apply because yeah but rest assured it’s there and millions of systems that people have taken heat to exclusive memory management from the GBM that might these is that I don’t think they need to do that much anymore yeah I mean if you’re just worrying about post times and latencies then we actually switched over one of our services that is customer facing it is dealing with rendering flame graphs and parts of the user-facing visualization data we switched it over to zgc we haven’t seen a post time longer than a millisecond ever and I think throughput Peak throughput went down by a couple of percent but we don’t care so simply not economical to like keep an aging set of object slots we replace the contents any anymore it’s actually worse in some situations but it was the fastest you could do not up until recently so I don’t

[45:54] Think people have realized how much the hospital around time has evolved when it comes to processing and bookkeeping data yeah but it also makes it much much harder to understand so Heinz brought up an interesting point with the GC logs and you know different GCS not being comparable to each other just comparing GC logs it becomes really hard to understand where time is being spent when you have garbage collection algorithms where the user threads will start picking up the slack if the allocation rates are too high and the GC can’t keep up it’s like that is also work being done by the TC right it’s just that your user thread is currently helping out a bit so it becomes you kind of need to look at the profiling data for the full system if you want to know where the CPU goes to at that point so this is quite interesting because a lot of things can be done a lot of things aren’t issues anymore a lot of things can be measured without fearing too much with the world so behind and Chris how would you sell this knowledge in a digestible way to customers how would you teach them

[47:06] To do an Apples to Apples comparison with without like overwhelming them with strange terms and complexity because I think this is this needs to be done in some way we can’t just say install this tool and everything will be magically explained to you I think we need to bridge the gap between like having data of equality we could never have before and understanding what it means I think it’s always a question of what is fast enough and what is the competition doing so if you’re competing against someone else who’s who’s who’s got a bunch of Engineers working at it to make it the most optimal fastest possible system yeah then you’re gonna have to throw some money at it but otherwise you just turn on zgc and if it’s fast enough you stop that’s basically it and these days again virtualization Rings Supreme it’s really not expensively die off bed a little bit more compute power if that is the issue of the face yeah just scale out a bit Yeah but if you dial up because you keep running out of memory bandwidth and it has the resources to power the entire world if you like don’t

[48:18] Hadn’t had that particular problem so you’re basically having the biggest servers in the universe to adjust for the single reason that you have a memory bandwidth is she goes away every time you add another 50 capacity how would you like from your unique position as as an analyst educator like server such a thing okay as I can tell you this compute power being wasted in the world today because oh yeah absolutely well that’s that’s the question if you look at all the systems in the world there are you know hundreds of thousands millions of systems who don’t it really doesn’t matter and you ignore those you focus on the ones like the online gaming and the fintech type of companies with the companies trying to organizations trying to process large data and stuff like that and with those you need to start start doing some some profiling but you always want to focus on the part which is the problem right so you and that all comes down to observing what is actually your bottleneck if it’s memory bandwidth you need to address that and profile that specifically so the

[49:31] Mistake that people do is they turn on too many switches to try and discover the problem with the profile it’s very hard to discover a problem with the profiler you can discover the cause of the problem with the profiler but not the problem itself so it’s like you go to the doctor he basically does a blood test for everything meanwhile you’ve had a broken finger you know that’s like you know their blood tests for broken fingers you know splint up the finger but that’s what programmers do he’s like ah let’s just turn this on and or if you really want to have fun go and ask chat GTP TPT what the course of your performance problem is and you’ll get a whole bunch of interesting results not necessarily worse than most programmers but still going in the wrong direction but anyway I’ll let Chris answer because I think more useful answers I mean I read a lot of the mailing lists for fun because that’s just the kind of person I am and there’s a definite Trend in the JVM to be more supportive of cloud environments and one of the things I’m thinking is that I mean previously

[50:38] Profiling was a technique for changing the difference between the system not working and not not being acceptable to being acceptable going forwards in highly containerized environments maybe profiling is now sold as a technique for reducing your Cloud Bill if you’re if there is a mismatch between the containers you’re creating and the jvms which are running in them and you and your simple answer is too we’ll start more containers we’ll spend more on our Cloud bill perhaps actually Performance Tuning is going to become a cost-saving function rather than purely a is it working at all is it working acceptably something like that I mean we’ve got we’ve got grav VM now as as a as an alternative to just in time compilation we’ve got project Laden the shape of the JVM does appear to be changing towards perhaps more but less less dynamism perhaps and more fitting into containers I mean I don’t know if everyone else is getting that feeling from the changes that’s coming out of Oracle yeah of all the geared that

[51:45] Feeling and from other vendors and providers and Tool makers too I mean I know Azure have a are thinking about like Cloud aware VMS and not just things like CNC sharing already compiled methods over the network I mean technically that’s what Gradle does for Bill Larry facts too so I mean yes we’re definitely moving to a more Cloud aware world but for some reason I would have expected that the JVM had more of this it already so that’s actually very interesting point yeah you’re making from Jacqueline the website which analyzes Java command lines I keep a there’s one of the pages shows you the last like 10 or 20 command last people have checked and quite often I see people with 20 to 30 switches in the in their command lines that the Jacqueline analyzes and says well actually these switches no longer exist in the version of the JVM you checked against or these switches are going to be obsolete it expired soon or the or this the defaults for this switch have have changed significantly across across different VM versions so yeah as as you mentioned the

[52:58] Cargo culting and copy paste command lines there’s a knowledge Gap which may be addressed by better ergonomic systems combined with the latest JVM enhancements but at the moment yes we do have we do a lot of people in the world with very long command lines but one of the things in the you know the Microsoft jet there was a statistic that 84 percent I saw 83 percent of the analyzed Azure spring apps did not use any JVM flags at all and so they’re not even they’re not even trying to size the Heap we should really like out of the boxer economics have gotten better and one thing like one of the most common Korea called command line nightmares I see right now is someone says like 15 years ago decided that we need to limit the amount of GC threads to two and then you’re looking at like the Box you’re running on and then you just remove that flag and the results are quite amazing and if you compare them just like out of the box so I think that’s like a lot of corticulting like that is still wasting electricity in the world we have a lot of customers running Java

[54:11] So I get to see kind of everything from customers trying to dedicate to 0.8 course to a Java application and being concerned about startup times to you know customers having thousands of threads running on gigantic boxes one thing that is interesting to me is how many old versions of java and really old ones you know thousands of CVS ago that are still up and running it would be amazing if you published a portfolio of your like pathological cases however at times where technology is really available and like that wouldn’t even see this as a problem if you just switch to that or if you reconfigures that like because yeah I found it really educational when I can point out that like an instance of this problem I’ve seen this before I sold out last gig and was solid like this and I mean that that’s something I feel is missing to quite an extent today yeah it’s it’s really interesting to be in a position where you have access to not just Java level metrics but you have systems metrics so that you can see other details about the runtime environment

[55:36] Together with the profiling data that you get from the runtime I mean startup time is not the single like future not proof thing that you should worry about these days no no no no no well we’ll probably be enough or today like suspend that thing rehydrate that thing I’m sitting there and hacking build caches and compromising the dynamic nature optimizations in the runtime or figuring out how to store like online state where you can basically use mem copy a running machine it didn’t just like a resurrected later it’ll be none the wiser that is an example when technology works for the Java platform and people tend to be using to they want the Adaptive optimization capabilities of the normal job runtime they want Peak Performance so we’re seeing we’re not seeing that much use yet of cloud VM some customers are playing around with it but it’s not that much and we are but yeah the solutions to this problem is definitely an area that that’s not inactive right now people are interested in this data and people tend to be using Java in ways you wouldn’t ever

[56:51] Imagine so I remember us discussing a long time ago whether or not the exception profiler for example should have two modes one capturing errors and one capturing all exceptions so that you could at least safely turn on the errors right so that you at least could get those and the Java language specification says that errors are typically unrecoverable things like your ran out of Heap right you can’t recover and you have no idea what’s going to happen next you will probably need to allocate something or unsatisfied link or your you know you try to use some some native function you couldn’t get get that you can link that up so we decided that yeah that one should probably be on always and that made sense at the time but of course somebody started to use that as part of a parser so for control flow kept and the error was called look ahead success and it was inheriting error of all things so we turned off on this early early days of the early continuous profiler data we turned this on we started to get billions because it was part of elastic that I think yep and

[58:10] We’re starting to get billions of Errors coming in all of them called to look at success so you can’t really trust the use of this either right you know if people can do something they will do it so you have to throw out all assumptions we’re never going to get to the best practice what it was designed for and but here I’m gonna I’m gonna throw this point out there and it’ll be interesting to hear from all three of you if you agree I would say that given the advances in technology these like 20 years of java a lot of the stuff that really matters to your performance the runtime operating systems and environments you’re on their orders have magnitude more forgiving towards 90 percent of the bad things or slightly bad things that you do see they don’t matter anymore as is the is this a Luger’s thing to say or am I correct it’ll be interesting to hear your opinion about that people still do bad make bad choices in terms of data structures they still make bad choices in terms of algorithm choices talking about someone writing something that’s in order and then and square instead of ROM I’m talking about

[59:22] Like all the 100 different things that really like added up to crappy performance like years ago I don’t think I totally agree I mean the JVM today does a pretty good job I mean we can still talk about like if inlining heuristics could be better to help more with scalarization etc etc etc but you know it’s it’s doing a fair job I would say at this point yeah that’s interesting well what do you think Chris Hines do you think that it’s it’s easier not to like have to do a lot of tweaking and scrubbing and cleaning of stuff these days we’re running on Modern jvms I mean are there things that obviously matter anymore but people that did spend time on because they mattered 10 years ago I think the Improvement in the GCS from early Java virtual machines to the current ones has has helped probably the most I mean the jit has got better there’s a couple of interesting projects out there to improve Escape analysis going on at the moment as well but I think the improvements in GC and the new GC algorithms which have

[1:00:38] Been added since early days of java have done a lot to eliminate some of the hackiness that was required to get the performance yeah definitely and they begot or they had to rely on the JVM itself being able to run stuff in parallel updated but gave me in parallel and in general you really don’t have these big likes around stuff below the surface anymore when you need to update pointers you compute things do you do stack walks thread local data and you can realize through locals don’t like their runtime internals have gotten very impressive as as like by necessity to them to implement the new GC so I think that is to me that looks like yeah there’s a quite a bright future ahead I think with all the new tools and new things we get we do have to sort of make sure we use them correctly like for example virtual threads and project Loom amazing technology absolutely amazing and what I like about it is how broadly it’s it’s been implemented in the JDK from you know rewriting the whole of the reflection API to updating all the networking and actually I was

[1:01:57] Almost amazed that everything still works even though they’re implemented everything normally you’re going to break something but they did their dog fooding yeah they absolutely did and it actually works it’s amazing so and the thing is with what I’ve often often noticed like I’ll give a talk at a conference and some people come after and say yeah but this is all like you don’t need to do any optimizing because in future we’ll have virtual threads and everything will be fast like well it’s left to use them in the right way you know so it doesn’t just you know virtual threads aren’t just faster threads you know they’ve got a certain purpose customers who think they can push a button and whatever they’ve been like whatever happened it’s gonna be like 100 times faster you know this I think is one of your more important professional Nations that you explain like what could be like I try but it’s it’s like my father would say farting Against the Wind because he is it doesn’t matter what you’re saying and you know with the funny things whenever you mentioned synchronized and virtual threads for

[1:03:07] Example people’s heads explode and say that doesn’t work I mean of course it works you know yeah there’s a lot of ancient wisdom people think their instances are yeah yeah and it’s the same with the flags you know you have thousands of one a thousand remember 800 Flags just with the JVM and most of the time if you don’t don’t say anything you’re going to be better off than if it’s something it’s always my first first step is remove all the flags and you know then we start them you know see if the program actually fixes it other apps with the same copy pasted flags and they have to go through there yeah exactly feels pretty overwhelming or stack Overflow you know what should I use you know it’s set it up like that yeah that’s a good point too yeah sometimes some of you as stupid as somebody in their code calling system GC yeah just to help along and that can be the bottleneck of your entire system the system GC yeah yeah that’s that’s on the same track as as people relying on the order objects are returned even though it’s not specified and then that changes exactly

[1:04:19] That you’ve broken our product and exactly exactly exactly how much time being spent on there let me try to wrap this up because it’s been an interesting performance discussion and we sort of agree on where technology is going what’s happened these 20 years and I mean it’s really hard to get that fully utilized and there’s a lot of Legacy apps that are never going to utilize and so on and it’s so much easier to get like Dynamic compute power for free and like in the light of how much easier this kind of thing is to do and like just grab another virtual server do you think our work is becoming less relevant than before because I mean let’s face it if you want commercial deliverables you can usually tweak it pretty well without knowing too much about performance analysis you can put some makeup on a pig and like for most purposes and dim lights people will think it’s that’s a beautiful Pig ah it’s it’s when given the ease and abundance of compute power that the runtimes themselves are getting better without having to think about it and so on I mean will things work like we’ll

[1:05:29] Get into a state where we not needed like at the level we are because people will be able to do their commercial deliverables most of the time anyway or simply just Brute Force up a little bit more computing power and it works or what should we do to be like to like all these skills that we require how can we best apply them for like come here I would say we really aren’t needed for most of the systems because if you look at how many systems they are and the number of systems we have helped is microscopically small compared to how many systems they are so for most cases we are completely unnecessary the VM does a good enough job and it’s always good enough that you know but there are companies who want to get the last ounce of performance out of their systems I want to save on their bills on their Cloud bills and for them it’s worthwhile to engage People Like Us I think that’s relevant these days to any extent I mean of course it matters when the Linux kernel is patched and the AWS Bell goes up 20 as as with models Vectra bug [Laughter] do they really care that we we’re

[1:06:45] Generating like 50x in Revenue that then with like the AWS bill is why would we be interested in doing like a find a local maximum there that was suddenly like it’s it’s effective 100 instead what are people interested in that I don’t think that’s not ready not many I’m saying not many I mean some are and some are willing to spend 65 million dollars a year on datadog for example but not many I would bring a different perspective to this because we’re in a different macroeconomic climate right now and people are caring about the cloud bills more than ever so for you an investment bank doesn’t care about the cloud Bill if it’s like if the revenue is 500 times sure sure but for example at datadog we wouldn’t have to sell the continuous profiler outside of data dog at all we’re saving enough money to pay for the entire team and that’s a data dog because of the scale that we’re operating at so you know that’s actually a very of course we sell it and it’s making you know a lot of money for data dog as well but we wouldn’t need to do that it’s

[1:08:09] Actually saving money and not only that since it tells the unblemished truth it also tells about the performance of the tracing component for example so as soon as we get a new platform where the profiler is available the Tracer all of a sudden it starts using much less CPU and much less memory because it shows up in customer profiles and we need to do better so you know and where performance is important we are always going to be necessary or profiling will always be necessary yeah but I mean it’s interesting to look at these people who still pay for these kinds of a product it’s relevant to them I’m interested that demographic is shrinking or like when the Tool gets better and faster and give you more info is it growing or is it just like the we really need to know is going on cases yeah it’s it’s it’s also used for and I think we we’re losing sight of this in this discussion every once in a while that it’s actually used for Diagnostic purposes as well it’s not only performance right so customers are interested in knowing how the performance profile changed from

[1:09:25] You know the last version to this version and they really want to know if they are going to get screwed on the next Amazon bill because somebody made a mistake and they want to know that before the end of the month when they need to pay pay the Amazon bill they want to know that immediately you’re still relevant obviously so Chris what will you do to be at Naturally useful for the rest of your career I’m lucky enough to work in a domain in an industry where it’s unlikely that performance is going to go away as a as an issue I work with real servers and Lease lines and high event rates so I I’m confident this still validate my point because you you’re still on the physical and not the virtual world and I think that is still relevant and you can’t just have the virtual world without physical under it so that’s a very good point that was meant as sort of a closing discussion but it seems to be that the consensus is that we have acquired a lot of skills watching Java and Java performance and mutate and profoundly change turning into like a server-side language from being

[1:10:37] Something for applets in the beginning or something from remote controls in the beginning I guess we’ve all pioneered technology there’s actually running in production today and it’s made the world a little better and I mean obviously some of the things some of the little boring things but not all of them aren’t as relevant anymore but it’s certainly a lot of evangelization and we can do software to build perhaps closer to the system than the application like the more and more time we spend in the space and the more it transforms but yeah it would be interesting to have a longer discussion of this say J Crete like where do we see the sweet spots where we can be useful and is it new exciting technology or are we more and more just applying classic knowledge that some have like slipped out of the product robot because it wasn’t relevant it’s it’s an interesting space to be in and for anyone interested in the final recipe’s job performance Chris’s book is obviously a very good place to start I was delighted that the first time I saw that someone actually had taken the

[1:11:50] Massive effort required to put this on paper yeah I’m still I’m still sad that I didn’t didn’t jump at the opportunity because Ben actually invited me to join the team to write the book those I was so busy at the time I said I’m not done enough time for that and that’s one of my biggest regrets of my entire life as that’s it a ton of Time Well Done Chris and Ben and James good job just wanted to add yes I mean Ben invited me to join his book it’s it’s really his his baby we should definitely give Bank credit for it for absolutely visualizing this project at not feeling completely overwhelmed like the story you put stuff down what page because when you go into this domain and you want to summarize like 20 years of expert knowledge in a adjustable form that is quite like it seems quite insurmountable when you begin something like that so I think it’s amazing we should try to contribute more to stuff like that and the challenge also with the book on performance is that usually the books don’t age well correct because of the way that the things change so you know to actually

[1:13:04] Spend the time and effort to write a book it’s like well done very good good job well I think it’s pretty like there is truth of that you have to keep it and keep it relevant but I mean like half the world is probably still range you have eight so yeah yeah exactly well not half but maybe maybe about I would say about 20 I don’t know any insights on that Marcus yeah I did have the numbers in like various organizations and markets but I don’t think it’s irrelevant anymore and I’m not sure like how much ndas I would violate if I’m probably none they all have to be safe right here yeah really I can’t can’t do a proper estimate as so I don’t want to guess but I know that it’s it’s quite there’s one of the more problematic things my job performance is usually a huge effort to do a tech uplift if you haven’t kept relevance and like with new it’s actually well if you spend the effort to get your 17 that is actually not that terrible to keep keep up to date and some some Works have the Paul like most of the fear of security which is of course the main concern they do want to stay up to date

[1:14:21] And they do need to know their future safe and for instance when I worked in NASDAQ they invested a lot in knowing that they’re not going to have any nasty surprises when third-party apps like stop working on older versions so they wanted to see that but again I think in a lot of places that doesn’t matter that much for whatever the business model is but yeah and it’s it’s I think so keep up to date that’s a good listen well this was it was quite interesting and I guess that like we geeked out a little bit a couple of times but I must say we not as much not near as much as I thought it would so Frank we’ll do some creative editing and if this is too inaccessible and thanks a lot for the opportunity to participate in this and also if you’re our speakers today Chris Martin is nice we really appreciate that you also in volunteer to talk a little bit about what it is you’ve what it has even learned for the last 25 years of OpenJDK