Foojay Podcast #43: Modern Java Testing

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] Hello and welcome to this new edition of the Foojay podcast in this episode we talk about testing how do we make our code reliable and what is a good test strategy welcome to the FJ podcast all your news about OpenJDK as a developer we all want to write bug free and high quality code writing tests is a crucial part to achieve this let’s explore the part of ensuring robust and bug-free code in the Java ecosystem whether you’re a seasoned developer or just starting out there’s a lot you can learn from the experts in this podcast about testing methodologies tools and best practices all empowering you to write reliable Java applications my name is Ronnie and I’m the founder of dma AI which is a free intell plugin for continuous feedback and test and I’ve been kind of working as a developer and Str struggling with many of the questions that we’re talking and we’ll talk about today for almost 25 years but so I come both with answers and questions U maybe more the latter than the first but we’ll discover more of that during the talk hello everyone I’m I’m sof developer and I met Frank and Ronnie

[1:33] At the local J where I gave a talk on testing the same one I gave on deok last year I also dipped my toes into blogging last year around the subject of testing and that is I guess why I’m invited in this podcast hi my name is zik and I work as a developer Advocate at Docker I work on the test containers project and it was a part of a recent acquisition by docker of the atomic D startup was a developer long time ago and then I converted to the marketing side of things but being exposed to task containers and working with Community I think I can contribute something to the discussion on the testing practices and I’m Frank Lor a Java developer since over 10 years and document documentation writer for aul also the producer of this 4G podcast to learn everything which can be learned about Java and its ecosystem and all the tools that are available for it okay where do we start with testing maybe the testing pyramid is that something where we can start with because I think it’s known to a lot of developers as the basic of a testing strategy but is it still correct it

[2:52] Might be time to reconsider it and the reason that I say that is that technology landscape is not as it used to be it’s kind of like configuration management you know when we had big large servers we had to invest a lot in the puppets and the chefs and all of these config management tools because infrastructure was costly in terms not not just of price but of time and there was no real option to just spin up a server on the fly with the you know to work with immutable infrastructure and therefore we all invested a lot in config management tools it seems that today with containers test containers and in general I would say the friendliness and the kind of support within the Frameworks for testing the cost of spinning up more complex test has reduced dramatically so in the same way that we can afford to let go of config management today and work with immutable infrastructure in a much simplified way working with or starting an integration test today is not what it used to be and for me as a developer that means that if I need to make the trade-off between writing a very thin

[4:21] Unit test that will test maybe logic and or very specific narrow logic and writing an integration test that can test more along the way punch for the buck today it’s not I would often choose the integration test which completely goes against the you know 80% unit test Paradigm of the testing pyramid are in your presentation that you gave also at our BJ you took another approach so you have instead of that pyramid with the unit tests at the bottom and then service test and integration test on top of it you have a different way of visualization the test strategy how what is a better approach according to you better I’m not sure but different for sure for first I didn’t come up with the approach but it’s called the testing honeycom I think it came out of Spotify Labs and it’s basically it’s not a pyramid anymore but it also consists out of three layers and the main layer is not the unit test it says you need to focus on behavior and to a lesser extent to implementation details and to an equal less extent to integrated tests but before we dive into that I

[5:49] Think it’s interesting to talk a bit more about what Ron said because it’s indeed something I noticed the technical Frameworks and infrastructure around the way we write tests has dramatically changed but also the way and the type of applications we write today in the past I used to work a lot on big monoliths with big domain and everything was inside today the landscape on my current project consists out of I think about 15 microservices and if I look at those Services the domain aspect of a single service has become thinner where whereas the integration aspect the adapter layer in a hexagonal terms has become bigger and more important a lot of these services are just basically transforming data so the there’s a shift coming on there that’s also yeah helped us move away from the testing pyramid is there also a big influence of test containers in this change of how we test things and I would be very glad to say say that yes test containers of course kind of spare headed a part of this Migration by giving developers API to do things in containers much much simpler and integrating both with

[7:16] Testing libraries and Frameworks to simplify connections right but also with the application Frameworks to simplify it on a larger well sort of service scale right if if you write an application and you have the full support out of it then of course it will be easier for you to adopt the approach and yeah indeed I think this transition of the mental model from like a testing pyramid to honeycom we very often advocate for when we talk about why test containers is a such a great library because it allows you to create tests at a higher level but in general if we take a step back from just thinking about in pure theoretical terms right this cuz this is the like a testing pyramid or honeycomb it’s an it’s an abstraction right it’s it’s sort of a guidance system where we would say like o I think I should maybe have more tests or maybe I should have tests and that there is a question right what sort of tests do you want to have and as a developer it’s very easy to have a like a card with an answer oh I need unit test for these components I need

[8:43] Integration test for that but in reality I think there is no there were very many discussions online but I feel lately in general there is very little Duality between those MH right the unit test the term was coined many many years ago right and then the unit is in there is originally a unit of computation it doesn’t doesn’t necessarily correspond to a particular class or a particular like language construct or even a component of the system right it can be can be anything you can call unit like the whole subsystem or your whole application so through the years maybe historically we were taught and maybe maybe by Academia Maybe by Just how the industry turned out we were taught that unit test should should focus on like individual methods so maybe maybe individual classes at most and you test this internal represent internal computation that is happening but originally the idea was just to take a slice of your system right quality unit run it so and then integration test is now when we say like an integration test when multiple components of the system are involved

[10:11] Right when we talk about test containers approach that is your application and then services around your database maybe mocks of your other services maybe your message broker and so on but it could be it could be just components of your system that normally don’t that are written in different ways right say just in time compiler and a garbage collector you can unit test just in time compiler just fine but it needs to be aware of the garbage collector and it needs to like for example meet some barriers or do some changes things differently depending on that so you want to test a combination of those and that that’s sort of like you can call that an integration test because it doesn’t test components in isolation but say for a normal Java developer would you make a distinction between between a jit compiler and a garbage collector being different components or not it’s unclear when I’m writing a spring application 200 stack traces levels above main that’s it’s not a dig at Spring it’s just a like with a very high level abstraction right I don’t differentiate between D and GC I just

[11:22] Say it’s my JVM so in my in my opinion this is something that I’ve always not always for a number of years now try to kind of ask people so imagine you have an axis of tests and on one hand you have unit test right and you can see see it and you say like yes this is we’re testing like an addition method a plus b both integers it’s clearly a unit test right and on the other hand something that is clearly an integration test right it spins up multiple processes it like gets your database a cluster of cfas hopefully with t containers CU it’s convenient right so we would like but anyone any developer would look at that and see that is integration test right M where is this distinction where is the border yeah can you where is like so and then the complexity sort of Rises right from one to another but can you say that at a certain point right there is a switch I would argue that you absolutely cannot I think this is the fault of conventions because similar to the way we kind of fell into the convention I remember was when ORS came into being it was all about Crow you have to have ad you

[12:48] Know people were talking about resources in h in HTTP and it was no resources have to be crowded in the same way with testing you had a method you had a test method you had had a controller you had a test for that controller test and it became almost ceremonial rather than about anything that had to do with where the logic is and because of that we ended up because of the template almost I feel that we ended up as you say that it was very superficial because it was very template based and people would use generators and would get the scaffold of these folders and it would say well I added a method I have to have a test was the name of the method I remember following that practice even and even then it made no sense and I think what for me was a little helpful in getting getting us out of this I would argue anti pattern of thinking about code as in any way significant in def finding your test plan the code and its structure was bdd because I think when bdd came along suddenly we got familiar with the concept of outside in and thinking about the behavior of

[14:12] The system and it introduced better practices through semantic conventions like adding should at the name of your a test and again it’s it’s just one way to do it but what it helped developers do helped me do was to think about what is the behavior that I’m testing rather than which class am I testing or which method and so on so and I think you’re spot on about the maybe false dichotomy between unit and integration test I would argue that if you have a very complex scientific calculator that you’re testing even though it’s just codee and it doesn’t have any containers or Services the setup of the environment to test it might be extremely complex even though it doesn’t fall into kind of the U definition of integration test it might be just as complex to set up and you might have single service integration tasks that today for example with tools like test containers that make it very easy to do just add an annotation and you have a working server so why is that now not a unit test so I think the fact that the definitions are becoming blurry is a

[15:44] Good sign because it’s kind of dragging us out of this set of assumptions that we were kind of victim to about a very templatized way to think about about testing I think also probably code coverage tools forced a lot of developers into let’s just make a test to test this method so it is covered but indeed it doesn’t fit in your business model that’s a bit the story you also have in your talk Jonas already the name of the test is a completely different approach that you have taken yes indeed I try always to Focus on the behavior that I’m trying to test most of us developers don’t write Frameworks you write applications for businesses and there’s clear Behavior to be written and to be tested and I see a lot of developers and I’ve done it myself because that’s how we’ve been taught to focus on the code to test the code and what you do you start with unit testing you mock out all dependencies that’s what you get taught and then you have 100% or 90% Cod coverage in best case and then you’re a good developer and I’ve worked on many of those projects and I thought we were

[17:09] Doing good until we had to refactor things but the behavior didn’t change business didn’t came to us and say said like this need a change no it was us we were driving Technical refactorings and what happened is I noticed quickly that a lot of the test these tests were worthless I didn’t feel safe anymore in a way as a developer because I had to Del delete and rewrite a lot of test because the dependencies that I mocked out changed and when I started to realize that I thought that took me a while actually to actually stand up and say like I think we’re doing something wrong because everyone is doing it around me so now I like to think that we when you explicitly test Behavior you’re going to implicitly test almost all of your code because else there’s Behavior that’s not used and the only cases that I found in my current code basis is like when you need to catch a checked exception or something like that is very hard to test from a behavioral point of view and on the other hand I also think that you cannot test an application only by testing Behavior sometimes you

[18:31] Need to test implementation details and what is the difference between an implementation detail and a behavior sometimes it’s very clear but sometimes it’s a very Gray Zone there’s no dividing line that’s also something I think is not 100% correct in even in the testing hono because they draw a line I think it’s more of a gradient sometimes we’re in the gray Zone I like to avoid this discussions with developers around hey you this is not really a unit test or that should be an integration test I like I don’t think that’s really relevant and it adds a lot a behavior can be implemented and test it with a unit test or with an implementation detail test or with an integration test that’s how you’re doing things but what you’re testing you’re not testing in unit or an integration so there’s a subtle difference there so about the implementation details imagine when an invoice is paid that the receiver receives an email know we got this email service interface real Java interface inside the codebase and we make an implementation out of that and we’re using we’re using

[19:50] AWS and we’re using the AWS Service to send emails it’s called I think simple email service mhm so we make an implementation we call it simple email service and if we test the behavior we’re going to say when the invo has been paid then an email should have been sent and that’s that’s the behavior that’s what the business cares about if it’s been sent through AWS or not or something else they don’t really care the email should be sent that’s that’s so you can write a behavioral test for that just test the behav as I just said it if you take that implementation that’s that simple email service AWS implementation and you test it in isolation for example with a test container because I think they have a local stack test container for example and then you’re more testing and implementation detail because if tomorrow it turns out that AWS is too expensive we’re going to move back to our good old SMTP server that we have flying around here implementation changes but the behavior does not and if I run that behavior test it should not fail because that eil is still

[21:08] Sent I would argue though and that’s a great example that the options here are not between testing Behavior or implementation the options here are between testing behavior and implementation testing just the behavior each with its own set of constraints and I think this is maybe a great kind of point about the cost of writing tests and maybe we can kind of slowly turn from just knocking the bad practices of the past like Old-Timers and talking more about a framework for the future and you know I don’t think we’re going to come up with very definite answers just because you know each project has its own definite answer so you know if there was a manual somebody would have written it by now but I think these are useful terms to think about the problem because when I think about the tests I think about pros and cons and I personally feel there should also be active effort thinking about the test plan and what is our strategy around testing for each project if you don’t have a strategy then you know it just becomes the last thing on your mind each time and you kind of an afterthought but if you

[22:34] Do have a strategy then the question is in what ways do you consider or do you make the decision about which test type or which test to write how much and so on and this is a question I see repeated a lot so I ran into this question and Rit and it was one of many where the person was saying how do you approach integration test how much do you write do you cover a lot of happy pass or do you go very deep and nobody there are no answers right nobody says you know these are the tests that you need to run and what you mentioned just now really raises in my mind the one of the costs when I think about testing which is the cost of change so I remember like a while back I was trying to find this comic strip for the sake of this conversation I could not locate it was this I have to find it was a comic I don’t remember if it was XKCD or simply explained or one of these amazing Comics strips online comics and it was showing this group and they were saying yeah we’re going to accelerate development because we have test now and we don’t have to we don’t have to

[23:55] To do manual testing and it’s going to be much faster F and then they add more and more test and then like two months later this product manager says I want to add this feature and the developer said no we can’t do it would break too many tasks and I think this is kind of one parameter to think about test which is tests can be cement okay tests can cement Behavior they can cement implementation in the less you need to consider that as you write them because if you end up spending a very long time just rebuilding your entire test we because you added too many tests on something that was still in motion it was still coalescing into functionality especially in agile work then you may have either introduced the test too early or you spent a lot of time covering or working on things that are maybe not mature enough to have these types of tests and I think this is kind of one dimensionality that I think about which is what would be the cost of test in terms of impacting my change but I want to open up this topic because I think it’s it’s an amazing topic which

[25:15] Is whether if if we don’t separate into and you know we kind of chose the easy way out because he said yeah it doesn’t matter integration test unit test fine but what if if it does not matter then if we have today all of these options right we are fortunate enough to be able to just spin up you know a spring server on with a few lines of code we have we’re fortunate enough to spin up Kafka or rabbit or all of our dependencies and we have the kind of good trouble of having too many options so now what should be should guide us as we choose our testing strategy what are the you know in I think one of my criticisms of the unit of the unit testing pyramid was that the AIS of the unit testing pyramid I think was were to me looked a little bit off I think it was me pull that up and so the question is what are the in this kind of decision making that we’re talking about about which type of technology to use which type of test what are the right Axis for this graph like how do we choose and what are we prioritizing yeah you’re totally right

[26:45] That’s that’s also why my talk was called your tests also need some architecting something I see see a lot that a lot of senior developers Architects they really have an opinion and want to have an opinion about architecture about the application but that’s always for the production code and you see Architects senior deaths saying this is how we’re going to layer the application these are the boundaries they create architectural rules through tools like Arch unit but then the developers are totally free to do whatever they want on the testing realm one death thinks I need write lot of unit tests and mock everything and other thinks no maybe not and I’ve opened project and it’s it’s a mess of different styles and different opinions it’s quite interesting actually on one hand we’re so focused on architecture but on the test that most of it just go I have seen mockup Frameworks that developers hand wrote because they were facing a difficulty and I could not believe the level of complexity and the level of effort in some way it was inspiring in other ways it was you know

[28:09] I wanted to bang my head against the wall because somebody spent so much time thinking about this State machine mocking where it might you know it this is kind of just another cost that you pay for your test if you end up with the this whole Shadow system that is intended to mock your original system other ways that I see a similar cost again talking about this new framework of thinking about testing is in test design for testability and there I think we’ve suffered and we’ve been the victim of again a very bad set of conceptions I have seen projects that could be extremely simple extremely thin become this hodge podge of dependencies and injections you know I’m I’m I’m a big fan of di but Di everywhere becomes crazy sometimes just because they wanted to mock each service and be able to inject it necessitated design decisions that had a big cost in terms of obstruction and today we know you know I back in the days sorry for sounding like an old gezer again abstraction was older AG the more abstraction you had the more sophisticated you were like what do you mean I remember doing that in

[29:40] Interview questions candidates would come in and I would ask questions and the answer was always dependency injection that was the answer and if they couldn’t think about that then they weren’t sophisticated enough and today we know to also favor simpler designs that don’t have that cost of abstraction where it’s very clear what’s going to happen when this code is being called and testing and design for testability was often colliding with Simplicity it was making you because you were looking at the internals of the code it was making you make changes and I think in that sense what test containers are helping us to test containers and any technology that allows us to kind of think create all of our defend from the out side rather than starting to inject services and create interfaces around everything along with other tools within spring and within other Frameworks is just it allows us to not spend so much time just working to change our architecture just so we’ll be able to test it and not change it for the better so I think another if we’re kind of thinking about the new

[30:55] Lay of the land and testing so another dimensionality for me for testing is what is their cost in terms of my architecture so are do I need to make my system worse just to support this use case called testing and sometimes it’s worth it sometimes it’s not the whole mocking thing I’ve been doing the same thing I’ve been creating monster mocks in the past but isn’t that because we didn’t have tools like test containers it’s because you didn’t know about the tools like test containers yeah yeah it’s a fairly it’s a very it’s a fairly mature old project especially in the Java space right it’s like eight seven eight years old right but yeah very often we did mock things and then we changed implementation and mocks would break and then that’s horrible but coming back to what Ronnie said right why are we writing tests in general right that’s the question like it’s an obvious good to say that oh we want change to be easy right like like it’s a it’s a it’s a true correct statement it’s just it is correct it is very important and if you forget that you might be misguided but

[32:24] I think what it what when we think about it what it means that tests are not just a sort of a quality assurance thing right we do use test as the signal of confidence how our application behaves ideally our test F should be comprehensive and complete right it should if it fails it should mean that something is broken and we need to address that not not something somewhere in the world is broken something is broken in our application we are working on right and that is like a lower bar to clear sometimes test are flaky and everything so it’s not the easiest thing in the world but it is a lower out of two bars right but it’s also for the confidence like thinking about test as the quality confidence tool the green test should mean that your application is correct that it works right and then there could be nonfunctional requirements for example there could be like performance issues or I don’t know like U interrup ability issues or something else right those are separate concerns and but your main test suit right should give you the answer what the application behavior is correct and

[33:58] Then and then you want to have tests from that point of view but as a developer and this is what yonas said I think it resonates with me what you said very much is when I as a developer write tests individual test not not not I’m not looking at the whole suit right I’m I’m looking at the test for a particular feature or particular bug fixs when I need to reproduce something right I think of tests as a developer productivity tool I’m not trying to enumerate all the use cases that my application has to conform unless I am in a very strict environment where that is truly necessary but then you know what you’re doing and like you are fine without this podcast but in general right I think I look at the test as a developer productivity tool I want to safeguard myself from accidentally breaking other people’s code so I will run tests that already there assuming everyone is the same on the same page with this and I will also write tests that will fail if somebody would accidentally break stuff that I wrote and for completely selfish reasons I mean like I do care that the project

[35:14] Succeeds that we have code quality levels and so on but my most basic motivation is that I don’t want to be the one that kind of digs in the code six months later when somebody else breaks it by accidentally changing other components of the system right I want to cover enough path and enough behaviors with enough attention to detail right and potentially going into implementation details to know that when somebody breaks it and the test that I’m writing will fail that will mean my stuff is broken right and then it would be either me to go and fix it which I don’t want to do especially six months later or it will be onever whoever is currently working on that particular change and accidentally breaking right so it’s a it’s a developer productivity tool to simplify change and then the I don’t even think you need to like come up with a particular architecture for the test it could be different for different projects but the IDE in my head the ideal situation is it’s very easy to add tests right you want the system and probably it could be your task as an architect in a project or maybe like

[36:34] A very frequently nowadays platform teams they can sort of advise certain ways of do things right so adding new test for functionality should be the passive the least friction I’m not sure if that’s like they could be a catcher more correct the pass of least resistance right the pass of least resistance so as a developer I would just do that without thinking and then me doing thing without thinking should result in the correct thing for say 80% of their time right so I want that in my project I want that in my company so everyone writes test and then when they write tests they will encode some business behaviors that they need to achieve if they need to go deeper they can go deeper they don’t have to in the example with the email server I would probably use local stack I would probably use the particular implementation and then switching providers doesn’t happen bully right it’s just it’s it’s usually a big decision probably will be warranted to change the test as well doesn’t doesn’t I don’t need to like reinvent the very high level behaviors where I register

[37:54] Gmail accounts and expect the emails come there right which I could right which is because it mimics what I would do as a manual quality assurance person right so but test as the developer productivity tool to keep my flow to keep me in the zone because when I break something right I don’t want to think about the whole system right I don’t have capacity in my head I’m I’m I’m I’m a very average developer at best I cannot do this right that I want that context of everything be externalized right and sometimes some on the lower level like language level we very value this right we’re like oh this language avoids us to emit semicolons that is amazing I don’t have to think about semicolons and we naturally are like yeah this is great and then but it’s a small thing right but we can see how it could benefit it’s like yeah sometimes I forget or sometimes there is or the other way around some people like semicolons and they’re like it gives me Clarity right but like either way we are like I don’t want to think about this right so with the tests I don’t want to think about other

[39:08] Components when I do something that breaks them I want test to tell me I want to reduce my cognitive load so I can find the place in the code very very easily right if I want to find how can I safely change or what do like what does a system do in a particular place as well right the opposite I want to change it I want to see which test break and I want to go and read the tests and see what is the behavior of like how might change sort of in a weird mutation testing way right explore the system I want that from the test seat so and the quality aspect comes naturally as a major benefit but to me as a developer could be slightly secondary so in a way they’re they’re executable pieces of documentation yeah you don’t want to be confronted with a failing test and you just simply read the test method and you have no clue what’s going on and then you have to read the whole method and then it’s hundreds of lines long in worst case to understand why it’s failing or just ignore the test oh my goodness you didn’t say that oh my God points forone my pet people people run tests and they start

[40:31] Failing and they’re like o this is fine it is CI it like always fails in Let’s ignore that in CI we’ll just run this locally or the opposite that’s another cost by the way flaky test and I’ve often seen flaky test just ignored so there was a there’s an intricate bit of timing or what not that’s very hard to capture vas and after you know people get tired of this test some sometimes failing so either you know if the build is sometimes red and it doesn’t mean anything then the build being R doesn’t mean anything at all and then that slowly kind of deteriorates into a situation where people start commenting out tasks I’ve seen that with performance tasks people just raise the bar because we hit it too often you know it’s the you should always be careful about the easy path out but I want to talk about I love what you said ol about the personal angle of testing as a tool for developers because I think it takes us a little bit out of the cost side of the equation that we talked about before in terms of what is the cost of test that you should consider to the value of test and here I

[41:46] Want to both completely agree with what you said but also challenge one of the assumptions so I completely agree that testing from a personal perspective as a developer are a great tool to get feedback and I think feedback is kind of at the heart of it because as I’m writing code the there are two questions that come to my mind all the time which is the first is what do I know about this code that I just wrote and the second is when do I know it and tests are a great way to say okay what know about your code like if you think about it what do I know about my code I know I write a new code or I change existing Behavior I can run it in my local machine I can maybe trigger a few things and then I’ll continue to change my code and I will have to keep running it and the scope of what I know is what I remember to try when I run my code so debugging your code is a very very poor way very manual way kind of like manual testing but worse because you don’t have a lot of time for it and it’s not allocated within the project to know more about what the code that you just wrote does and how it behaves and

[43:10] Getting feedback is extremely important otherwise you’re really relying on your skills to just get it right in every possible situation for you know every possible scenario I knew people who could do that like I knew one person in my team he used to write code then he would delete everything write it again then he would delete everything write it again and the third time it just worked no need to test no need to compile he had this compiler in his head I don’t know how he did it he was a savant but we’re not all like that and even he you know eventually this Paradigm would fail so the question is if we have if we want to get feedback and test give us feedback the second question is what type of feedback can we get and this is where I want to challenge a little bit the Assumption because at the very basic level yes you could validate assumptions you could say yes when I add something to the bin should have this something that I added classic test right and that’s great this is kind of like basic sanity just to make sure you know all of my assumptions about reality

[44:25] Are still correct we’re still in business I didn’t change the nature of reality it’s really important as a developer to maintain sanity but there’s so much more because if you are in fact not just testing implementation if you’re not just testing a very narrow set of assumptions then you can get a lot of data out of test and for me everything is about data in this sense tests are amazing piece of data why are they amazing pieces of data because it’s kind of to it’s kind of like the scientific method you have an endless set of experimentation each time you try very hard to isolate all of the factors and except the code which is changing between iteration and this is amazing right how imagine the type of in we can get out of this type of experimentation especially if I’m running my system I’m running testing out all of the engines sometimes in different permutations in order to get that data and I think in this is one of the major opportunities of testing we’ll talk later about Ai and I I’m less optimistic there but in terms of getting more out of what we have using

[45:57] Technology and tools that are coming in so one aspect that I talked about earlier which I think completely changed the testing pyramid was cost in terms of slow I think slowness is what the testing pyramid talked about there is complexity and these things change dramatically with Frameworks and tools but the other thing that is changing is the value so we’ve run our tests now what so in the current world the now what is green red but if we think about test as sources of data then in the new world we can think about nonfunctional benefits from that hey look you have a performance regression hey look you’ve introduced 40 new queries did you mean to do that or did you just call a bunch of functions that called a bunch of functions that eventually created those queries hey look you introduced an n plus1 in your mapping so all of these things are things that are available in the testing data and the more you also gravitate towards other types of tests the more data the more diverse data you’re going to get the question is who looks at that data and unfortunately right now it’s

[47:18] It’s it’s kind of hard to convert that raw data into insights and this is not just about testing I think this is kind of going to be with us in the development industry for a while where we are becoming amazingly Adept at Gathering more and more data and I think like observability is a great example because with observability today you know we can get unimaginable new insights like before when I ask these two questions what do I know about my code and when do I know it I could say well I know my code works I know my test Suite run and I know that whenever it doesn’t work in production I get alarm Bells today it’s different because the code is you know open Telemetry for those of you familiar with it or it’s very easy to enable and it spews out massive quantities of data now the question is what can that data teach us about the system and here is I think where there is an opportunity for testing to provide more benefit and this is something that I’m experimenting with a lot which is how do I create a test that over time lets me know more than just about the past fail which I take now for granted but also

[48:41] Tells me about the quality of the code so not just a Boolean true or false but it tells me hey you’ve made the system better congratulations you’ve you’ve actually simplified something this now is more efficient you’ve removed a critical section you know these are things that are hard to measure via pass fail and they’re not even there for like a Boolean kind of type of validation they’re there to teach you as feedback about the code because for me it’s testing ultimately is about getting that feedback back otherwise you know after you’ve gotten used to test you know without test you’re running blind and the more you get from Tales the more you feel blind if you don’t have that type of information back so you’re not mentioning your company name Ronnie but that’s actually what you’re doing with digma it’s a plugin for intelly which gives you all this information company’s called digma a you mentioned artificial intelligence I’ve seen different use cases related to tests it’s giving it some piece of code and asking it for a description that’s something you can do with

[50:02] AI probably I think intelly is doing a lot of AI integration into their IDE is write me unit tests for this code is that a good idea it depends on the on actually the answer to the previous question what do we what are we why are we doing this right why do we want to have tests in our project right if you wanted to assure quality assure the set of behaviors of your application then I think it’s a horrible idea right because if you wouldn’t trust the AI to write code in the first place M why would you trust AI to write a different code that would check the first code right so and if you do trust AI to write code do you also trust AI to actually write code that verifies the original code right and if you do what are you doing there yourself right so it’s just then supposedly it’s to the point where you can reduce the team dramatically because this is what the developer does they take sort of vague input some external knowledge that your like environment has and transform that into changes to the system that bring business value so I think the

[51:37] First answer is horrible idea and the second answer is I don’t think we are with the current tools at least at the junction yet where the trust in the AI output is so high right so I think what we will see first is developers or at least this is my I think a more sort of conservatively sensible approach right would be I would write my test I would codify the expectations and then I would be like AI introduce the changes right I have my confidence in my test right I created the system so I know that sort of test driven Behavior way but not exactly I know that is the bar to clear for Quality right and then I would be like yeah do the code right if we if we think about the importance of the source code or tests I would rank the importance of tests much much higher and this is something that historically I didn’t I didn’t think so very much but like more and more examples kind of point that way right for example if we talk about Java development right the JDK development is the JDK the most important part of that or tests for the JDK mhm and I would argue the second

[53:12] Right the latter for both some like sort of political reasons and Licensing reasons but also for the implementations right if you want to create a different implementation you test not just to call something Java right but to actually be a compatible VM so I would venture to say that AI would help first with a less critical part of code first and then we will get to the point where we can run we can generate tests so that’s that’s that’s my take on it I don’t think we are there I don’t think it’s going to be an easy path there so I think developers are largely safe currently from the threat of being eliminated as a profession however look if if you kind of see the impact that it’s been having what I’ve seen with many developers is that testing was kind of the you know lots of developers don’t enjoy writing tasks and the tasks that we don’t enjoy doing we and delegate so I’ve seen many projects where I ask them do you use Ai and they say yeah we just use them to generate tests and there is a lot of problems with this I think and I don’t mean to be audite like

[54:41] I love Technologies and I love seeing them make an impact but I think in this case you know we should always be careful of the easy way out and in this case if you say oh man my team lead asked me to write a bunch of tests so let me have chat GPT do it and they end up kind of looking nice then I’m done but there are multiple problems with this approach the first problem is that these tests and I don’t know Jonas have you had a chance to generate test with AI yet all developers have have made an attempt and it’s not happy with were you happy with the result no absolutely not but I did use it a couple of times to get rid of the toil of my job when I knew the answer and it was basically just typing MH and I could give thei system an example of what I wanted then there’s value in there because I knew the answer up front it’s just typing work it was a l typing I have I had a happy pod test and I knew I need five edge cases that I wanted to test that’s a lot of copy pasting and I just described my problem gave the happyt test and he gave me some yeah good feedback that I could quickly reuse

[56:03] What could have been a parameterized jit test you through GPT at I think that there is something tempting in using AI for that because it’s some for some devose it looks like a menial task the so the one problem that I had was that it did it wrong it did not you know consider any of the practices we just talked about it was more like the old test method approach unless you directed it to do otherwise like method to test mapping but let’s say they fix all of that the and you know the tool matures and it does bring you good insights or good tests the problem is that the process of mentally as a developer thinking about tests is important by itself so defining your assumptions defining your behavior is important for me to go through and in that sense my usage of AI like other fields by the way as well is kind of like to break my writer block or to get different perspectives I like to throw something at it and say describe the behavior via test and maybe I don’t agree with that but at least it gives me maybe some dimensionalities of the behavior that I did not think to test at

[57:21] All or so this is kind of taking the backwards engineering approach where you reverse engineering approach where you take something and you ask it to describe it which can be useful the other aspect is if you do ask it to generate some tests there is no way in the world you’re just going to check in that code right there is no way in the world that you you’re not going to be mentally familiar with these and that they don’t describe the world in a way that you understand understand and agree with and I think this is kind of the where there is a lot of thinking that has to be done that tests encourage and this just highlights the fact that it’s very hard to be to do without those thinking with without that thinking because eventually when that test failed and you look at that test and you say hm I didn’t even know that this was supposed to be the behavior then you’re in kind of doubl the trouble because you may have inadvertently had a green test but the wrong behavior and this kind of reminds me of the what does it mean that the tests are green why do we need green

[58:33] Test for dilemma which is if the test is green does that mean the application is working and the answer is of course no it does not mean that at all everybody can write a green test it’s the most easy thing to do in the world so if the goal was to have a green test you know it’s it’s it’s not a challenge the problem is can are the test accurately describing Our intention and in that sense we have to be involved so it doesn’t matter if we get a scaffolding generated by AI or not as long as it follows the right conventions I’m all for it like if it’s actually creating a good starter kit for us to do the rest of the work if it knows how to I don’t know I always forget like which annotations to add in which order what to do like it’s it’s like it helps but it doesn’t replace the thinking because ultimately this is what you’re validating and you’re supposed to know what it is so I think there is a piece there that cannot be easily replaced by AI just yet you mentioned at some point most developers don’t like to write tests I think that means that there is actually a problem with your

[59:54] Job that you’re doing I like the test because it proves that what I just did is correct or it shows me that I forgot to do something so it’s and also Yas mentioned a test is also documentation it tells you how the application should behave for certain use cases so if you don’t like writing the test then maybe you should maybe think about how your project is architected like y said how the tests are architected the tools you’re using using or you’re using the right tools so I think if you don’t like your tests then it reveals some problems in your project correct yeah I think you don’t like that project at all if writing a test is a chore then working on that code base is a chore like in general like I cannot imagine like a system where you’d be like oh it’s so hard to write a test but then you be like but I’m going to go into the deep depths of this code and randomly change things there and that would make me happy and productive i’ absolutely no right that would be even bigger problem I have a confession as somebody who wrote test Frameworks in the past that for different tools we

[1:01:16] Worked on as somebody who worked extensively on testing I don’t like to write tests it’s not that I it it’s the least favorite part of my job like when I’m and you know it’s it’s not that I don’t write test it’s just that but I confess that it’s not my favorite part like after I like I’m I’m all in working on this really interesting feature I figured out everything I’m working the last thing I need is to stop what I’m doing or even and start fiddling around with test infrastructure to get it set up and working the way that I want it’s not that I don’t believe in it I am I’m a firm believer and it’s not that I don’t practice it but personally it’s just not my favorite type of thing to do so should start with it I don’t get that you should write the test first no not always by the way I maybe that’s another assumption to think about like I know that many people believe that test should be written first for me it’s not always a good approach and I’ll explain why sometimes I do it especially when I’m describing a very high level behavior and it’s kind of very and in that in

[1:02:39] That case it’s sometimes makes sense but sometimes as I’m developing it’s where I’m a Believer in kind of this maybe Microline approach to development where you need to work through the problem before you can start working through the solution and working to the problem often doesn’t require changing a lot of tests it’s it’s it’s for me working with the code and testing it out kind of in a sandbox mode until it’s just right and in that scenario I don’t always go and update the test and update the code and update the test and update the code so I don’t always write the test first and I for me at least and you know this sounds maybe a little heretical it’s also not the favorite part of the job but having said that don’t you dare not write test and don’t you dare like not think about that aspect because that’s what’s keeping you sane what that’s what’s grounding grounding me as a developer the fact that I’m getting feedback about what I’m doing so I was absolutely ecstatic about the fact that I don’t need to spend as much time on test that I get I don’t need to

[1:03:55] Work as I did in the past about setting up the test environment so much that but you know as I mention all of that I still do a lot of Works for test infrastructure I still build factories and in order to create you know as much as possible using my own API so that it’s not as brittle but I create the entities that are going to participate in the testing I formulate these assumptions and the experiments it’s just that personally I love kind of being in sandbox mode more personally if you have tests in the same pool request without your reviewer asking where are the tests you were doing test driven development by me you were test driven enough I think it differs very much application to application or like sort of application type to application type Rony so what you’re saying if if if I’m working on a u a more mature code base and I’m introducing changes right then usually that means that there already an existing behavior that needs to be adjusted in a in some way that means that there are also existing tests where I can take inspiration for isolating the smaller bit that I

[1:05:21] Need to change right if I if if it’s a bug fix or a maintenance thing then starting with the test just makes sense you want to reproduce the current behavior understand the inputs map them to the actions to see the whatever errors or the behavior you want to fix or change right so there it’s a natural place to start with a test you can of course start not with a test by exploring your application first right but before you start doing code changes it makes sense to kind of sort of codify the assumptions right Behavior so there test comes first naturally and well it’s hard to say that whether this is my favorite part or not but it it’s just a very natural part of the job that where like I don’t I don’t like when I think about this development process right I don’t again I think of test as a developer productivity tool I don’t think about them so you know this devops like cycle infinite Loop right M I think it’s very much misleading in a sense cuz it has built like code and build and test phases separately right and that test phase there it actually signifies sort of like a c

[1:06:42] Environment this high level non-development kind of standard right like the single source of Truth for the quality and then you normally do that before it goes to the outer Loops of like development process right but sort of what the inner loop looks like to me as a developer it’s not this one part of the devops cycle right cuz like I don’t very often I don’t do the whole thing code build test commit right when I do code I run the test this is my this is my cycle I want immediate feedback I want I want to be in the flow I want to be in this Zone doing changes and there writing test and writing code changing code exploring what will happen when they change this right what are the implications does it suddenly change my application to go to the database like 700,000 times right that sort of like I don’t I don’t separate that in the separate thing right so when you when you think about this oh I’m going to write code and then when it’s all ready I polished it it’s ready to go to production right it’s ready to go into commit it’s like fully yes this I’m done

[1:08:00] And then you’re like H I need to write tests then of course it feels like a chore in that case all like then you already rot to implementation and you kind of know your test will be driven towards what you just implemented 10 five an hour ago or minutes ago but if you write your test first it’s it’s a totally different world because you have no impementation you need to think what is it actually that I need to implement so how is this going to behave what are the parameters what’s the input what is the expected output when is it correct when is it not correct that’s why it goes hand in hand you write like you never you like you don’t ship your first prototype normally like you can I just don’t want to get into this assumption that all developers first of all that writing test first is a must and second that most developers enjoy writing test because I don’t think that’s a good assumption no that’s not I think most developers don’t yeah and I think we need to kind of acknowledge that and you know I’m I’m I’m no different like I love writing code more than I love writing test does

[1:09:09] That mean that I will not write this no it does not and at the same time I think that we need to consider that we need to support or like it’s it’s very easy to get and I had this issue with you know tdd Advocates and tdd I met some tdd guys that were pretty fanatical about it at the time and they were doing tdd at the micro level so there were like every little thing every function needs to start with a test and if you think about the gap between like how much runway you’ve had with your implementation versus how much time it takes for you to catch up with a test for each G change they ended up with Suites of tests that I recommended they throw away like most of them were too brittle were too dependent on implementation I’ve seen and you know there are so many ways to get this wrong especially if you don’t understand or don’t keep that perspective that the behavior and the outside is matters more than the inside and what they would have is they would have the same functionality like the same code basically written twice essentially when you examine the test following this very

[1:10:30] Prescriptive process of writing test first they would have the code once in the test and twice in the inside the application implementation itself and it was the same code and if they had a bug it just writing something twice doesn’t protect you from having a bug especially if it’s an implementation detail so what I’m trying to say is that I completely understand and that you know some developers might prefer to be very let’s say disciplined about writing tests in a very specific way and Manner and maybe it’s the even more correct approach but realistically many of us are going to look at tests as something that is valuable but not necessarily our favor part of the job and to make that still happen we need to set to have the these ground rules about which types of tests do I need because if I don’t have that then I’m doing tests and again I don’t want to come off as being too you know yeah of course everybody has to be perfect no we’re not perfect and I’m sure there are many developers that complete their pull request and then they say oh man I have these tests to

[1:11:56] Write as you said or maybe they wrote some test first and then they got carried away in the inspiration of writing code and they didn’t come back to the test until the very end and so it Remains the question of what do the test that I include in the pool request what is their function what do I need to get from them as a developer and this is exactly the type of value that we’re talking about now so if I can get from the test more confidence in my pull request if I can know that I’m not ruining something that I did not anticipate then they serve an actual function to me rather than be a ceremony like if I actually got insights out of the test and the insights can be codifying my assumption which is to me is the least part of the job because it’s important it’s it’s great for keeping keeping track of changes and from kind of making changes possible but at the same time if I got more information out of them if I were able to know okay I was introducing this new function and I was as I was doing that the test helped me find issues that would otherwise I would have

[1:13:14] Found in integration in a later stat of the build in production then I gu something out of it that’s very tangible and ultimately many of us are kind of especially when we’re on a crunch when there are releases when people are pushing us we want to we’re self- serving we want to do what we can do just to be able to move on to the next task and in that sense it’s only by connecting developers to the real value that they get from testing that it makes it worth while for from them and I’m saying that because I’ve been struggling with this I’ve been talking to developers who said okay it doesn’t matter if they wrote their test first or not we have a set of tests we have the code we’re ready what did you learn from the test I don’t know I saw this so I wrote this I copy pasted another test I changed the name a little bit and it was basically the same so now I have this test it’s going to run and it’s going to be green on the next build now what I just described is insanely hard to prevent from happening at projects at scale from people kind of taking that approach because if tests become especially when

[1:14:35] The test project is a mature and it’s really just a matter of copy pasting another test method and changing a couple of things and you have another green test to show for as a trophy then you’ve missed the purpose but it’s very hard to F tune towards what the purpose is so unless I get actual value out of the test Beyond just saying you know should blah blah blah and then seeing that turn green then it’s hard to pinpoint exactly how much test and which test to write when you say test like this right I think a part of I don’t know if this is a subconscious bias or you sort of on purpose try to phrase this there is no decoty there is no deoy between code and test tests is code right so when you when you say when you say when you say how do I know what type of test to write right how do you know what type of code to write how do you know that you need to use a factory here or an interface here how do you know that at a certain in a certain Junction of your application abstraction is necessary or maybe you don’t need abstraction there right it’s

[1:15:58] Your it’s your experience it’s your knowledge as a developer talking the same tests are code how do you know that you can copy paste a certain test change values and that test gives you confidence that whatever un functionality is created that’s your that’s your worst as a developer test the code when you when you’re in the rush and you somebody comes to you and says like like you don’t need test just ship it right stand your ground be like I cannot ship incomplete code why is it incomplete because tests are missing tests are essential part of the code what is the definition of complete that’s what you define as a developer exact sign you sign off on that if you have performance in your definition right in your spec requirement right you better have a performance test you better have some sort it doesn’t have to be like a benchmark setup it could be it could be extrapolating performance from a few sample runs of other tests or something right but like you better have that data I want to Echo something here and this is really important this line is not clear to anybody so when people ask on

[1:17:11] Reddit how much test is enough do I need to follow just the happy path how many of the edge cases should I cover in testing they are right because testing can be an opportunity and it can be a rabbit hole and as a rabbit hole it can I can very easily spend infinite amount of time making the test more and more complex because reality is very you know it is it doesn’t come in straight lines and as a result of that there are many ways to wrap it to wrap around it was test so as opposed for me okay as opposed to stories that always have a beginning and an end in the way that they’re defined testing if you ask me giving a set of spec which test to run I can give you three developers who would have completely different approaches on how much test to write and I want to help get the right definition because if we or even the tools to make that call because if we don’t have that then the problem is that we end up with whatever time we had at the time sometimes can dictate to a developer how much to invest in that and this is why I think testing strategies are important

[1:18:32] Because we need to understand what is our goal in creating our test when do we know that we’re done that’s good point not disagreeing can I just lighten the atmosphere with a joke yeah right just to finish this off so the consultant comes to the team and they’re like oh I think test driven development could help you guys a lot right right so remember you write the test you see it fail then you make code to pass the test and then you refactor the code to be beautiful and optimize performance and so on so remember red green refactor red green refactor clear clear in two weeks they come back to the team and they’re like how’s your tdd right how’s your T development and the team is like yeah we’re doing it this great the only issue is we never get to refactoring part right and the coach is like why is that how come is that well because when the test pass our definition of done is met that’s a nice one yes a nice one to conclude thank you for your time thank you for joining this podcast definitely a topic we should come back to because there’s a lot of open discussions and open questions here of

[1:19:43] What is a good test what is a good test strategy thanks for your time thanks also to the listeners of the podcast keep an eye on Foojay for future articles about development and everything related to the Java World thanks for listening give me a the of OpenJDK