Foojay Podcast #26: The Future of Source Control and CI/CD

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] Hello and welcome to this new edition of the Foojay podcast today we are wondering what the future of source control and build automation will bring us all your news about OpenJDK as developers we like to automate the boring parts of our job The Source control system build pipelines and code analysis tools we use are crucial parts of this Automation and like almost everything in the software World these parts are constantly evolving we’ve switched from CVS to subversion and from subpurian to git and we started to embrace trunk-based development instead of creating branches for every feature that we work on so is this the end point did we find the Holy Grail in Version Control already and what additional Evolutions are perhaps still waiting for us let’s find out together with these amazing guests hello I’m Trisha I am a lead what’s my title now lead developer evangelist at Gradle I joined Gradle in January before that I was working at jet Brands doing IntelliJ IDEA advocacy and last year I had a tiny break to write two books because that’s what you do when you want to take a break so the

[1:16] Thing that kind of thread that kind of follows all of my work in especially in developer advocacy has been sort of developer productivity how can we use our tools to be more effective as developers how can we automate the way the kind of boring stuff which is why it’s really interesting to talk about Version Control and CI CD and things like that hi I’m Michelle Reese I’m at developer Advocate at jfroc this is my first time being a developer Advocate I have been a consultant for the last 20 years so before I just all about developing software and now it’s about sharing learning and finding new ways of bringing different paradigms into developers and also bringing back the feedback from developers to the different companies because that’s also really really important for us and I’m super excited to be here so thank you very much everybody for having me hello my name is I guess I was invited here because I’m the author of one of the main offers of people which is a version control system so we have a head server control system based on new ideas from mathematics it’s it starts as a as an improvement on

[2:39] Darks which was itself also based on so much for the theory of patches and we’ve expanded it to make it super fast and easy to use scalable so before that I was I’m working on this now before that I was a an academic researcher on a variety of topics mostly related to ASAP Distributing models before covid-19 I was working on a molecular Computing so how to build molecules out of DNA how to program matter to assemble itself and after it’s like since school videos and now I’m working mostly in the energy sector trying to help people share renewable energy with their neighbors this is also a an interesting theoretical problem because you’re you have a shared resource and you want people to get enough of like to feel that they got enough and then at the same time to optimize profitability but I’ve decided and I’m also working on people in terms with us so lots of things in problem thank you and I am Hollow Embers I’ll be your podcast host for today I am an I.T consultant at info support in the Netherlands and when you’re in Consulting like me you tend to get a lot of opportunities to work on different build

[4:00] Automation setups and virtual Control Systems which is one of the reasons why I developed a special interest in both topics so thank you for being here today lots of interesting angles I think we can take I think the one thing we can agree on here is that as developers we like to spend most of our time on actually writing code and delivering values to our customers right so that means the more we automate our CI CD process the better it is right or are there also any downsides in automating everything we do HL I wanted to ask this question to you what do you think about automating as much as we can of our CI CD process well that works perfectly fine if we are thinking about the happy part because then it’s a matter of doing the repeatable things and that we know the inputs we know the outputs and we know the process so actually by adding the human factor there we probably are adding more uncertainty and we will be actually the part that is going to bring the errors there are other tasks by the nature that they cannot be so easily to either establish or automate for example if we don’t have

[5:26] Enough knowledge or it’s so rare then the effort of trying to automatize that it’s it’s higher than the return of our investment and then are the tasks that are done to create new knowledge like we don’t even know what the input is or the output we’re trying something completely new so as you see when we’re talking about one side of the spectrum the ones that input outputs are known and the process is well developed we know what we want then for sure we should it’s our duty because that is going to remove a lot of the uncertainty of problems that we have with technology and most of their tasks are like that but no is everything visible to be automated no should we no I think I hope I kind of answer that question I know yeah sure you did because I fully agree with you so Trisha you told us that one of the fields that you’re interested in is developer productivity and HL already mentioned the human factor right well developers are also humans obviously so could you tell us maybe something about that how do you see the human factor working its way into developer productivity or at least maybe having a

[6:55] Detrimental effect to that greater we’re talking a lot about developer productivity engineering which is about automating all the stuff that can be automated basically removing toil some of the stuff that Michelle was talking about in terms of the stuff that’s repeatable the stuff that’s predictable that’s that lends itself well to Automation and so we’re we’re talking about DPE being the automation of the stuff that adds friction increases toil which means things like obviously automated tests automated deployment is where possible being able to do things like the tool that we have allows us to speed up our build with both Gradle and Maven so that the build doesn’t take so long running the tests don’t take so long all these things that you to end up waiting for as a developer that allows the human to do more stuff which is good that the human’s good at right so if you’re not waiting 20 minutes for a build time for your build and tester on you are getting fast feedback from your tools then you can be thinking about the next step and what developers are really good at is being creative so of course we type stuff into

[7:58] The computer of course we tell the computer what to do but the difficult bit is not that bit the difficult bit is how do I solve this business problem have I asked the right questions have I thought about all the angles have I thought about the way the user is definitely going to add the wrong thing and it’s going to break and machines can’t do that and although we’re coming a long way in terms of being able to do generate various tests which can throw things at throwing rubbish at your application you got like what’s it called like chaos Monkey Style from fuzz testing and all those things you still need a human to kind of make the decisions about what should the application do what does it look like when it goes wrong when is it likely to go wrong and so for me developer productivity engineering and which is aimed at improving developer productivity is about automating stuff that should be automated I talk about this when I talk about code reviews as well like code reviews you should not be looking for formatting you should not be looking for anything which a linter can do or

[8:57] Anything which an automated test can do a human is not good at that so if you automate stuff that is automatable then the human element comes in terms of Are We building the right thing are we building it correctly have we tested the right things have we covered all the areas that you know we wouldn’t have thought of instead of spending and I’ve been in jobs like this instead of spending three hours running a manual process to release to the test environment every day you get to spend those extra three hours writing good automated tests or having conversations with a business about the problems you’re trying to solve and the thing that a developer should be really good at is being able to take these requirements start working on things but then go oh what if and then being able to take those questions back to the business because those the sorts of things that maybe the users in the business haven’t thought about oh yeah it’s true that we haven’t really thought about this angle or you as a developer are thinking well we have a module which does this and you want a module which does this what if the interactions

[9:58] Worked like this how would that impact the business or the user and that’s what we want to free the developer up to do we want them to stop thinking stop spending time on stuff which is like high friction like low value and allow them to start bringing the good human stuff to the process yeah we tend to produce still some good human stuff right there is the human factor but there can also be benefits to that Pierre Chen do you recognize some of these challenges that a lot of time can get wasted waiting for builds or test results and how do you typically tackle them yes yes absolutely so it’s interesting to hear the industrial developer perspective because I’m not in this field of operating myself like I don’t work in a huge projects with very large teams and in a safe and safe people comment to you and say well the customer wants that and this and so on so yeah I definitely recognize some of the challenges so one one thing that came to my mind as I was listening to excellent and Trisha was that’s sure automation is important and good for you and good for

[11:09] Everybody but not just any automation for example one common pattern is that you have a problem that’s completely artificial that could be solved by just not having the problem in the first place and you’re like okay let’s automate that let’s automate the solution for example in developer developer development workflows sometimes you’re like well when you’re using the wrong tools like for example in the back in the day where everybody was using SBN or CPS some of the problems was locked on files and so people were wondering how to automate this the way to take a like on the file and then we all have asynchronous distributed Version Control Systems and we’re like oh this problem no longer exists so many times just you know being clever goes a long way before we actually need automation so that’s that’s one different perspective about sprinting table so it’s many times yeah automation is good sure but sometimes just thinking deep about thinking deeply about the problem you know modeling it’s really carefully and making sure that challenging it is important for example

[12:20] Sometimes you’re like you have a model and that’s the first thing that comes to your mind and you’re like oh yeah well that was really good it took me a lot of effort to produce it and your favorite challenges and you’re like oh yeah let’s go with that and then you encounter lots of problems along the way that certainly you can tackle with automation but also by thinking a bit more or challenging your models a bit more before before starting yeah you would probably not have even met problems and another comment I had about about something both of you said is that one of my one of my use cases was a French National Assembly so the thing that made me realize that was the comment on the human factor and what should be it should not be automated so the French national assembly is like any parliament in the world is a typical example of Version Control bills that come to parliament are very rarely new laws and we’re completely out of the computer program are real here right so bills are very early new laws usually they have written as a patch to existing law and they are really written

[13:34] In that form if you look at bills you’re like oh change replace words replace every occurrence of words two by words bar and that’s that’s like all all bills and all Partners in the world like most parliaments have ruled that at least are exactly formulated in the same with the same almost the same language or at least the natural language version of what we do with Unix diff and so you have these highly qualified lawyers and here comes the human factor right you have these highly qualified lawyers who take the amendments to the bill so change patches to Patches so they take these amendments every week and then they try to they’re trying to find the conflicts between the amendments then they presented then the presentation rules right so voting is like a trend-based development and then they manually apply them to otherwise they resolve the conflicts by not presenting to both amendments that would conflicts with currently voted amendments so while they’re in a session environments they have to keep track of manually keep track of what amendments were applied and

[14:47] How they were applying what that changes for the next amendments and so on that can completely be automated not not up to the last sentences where people have to actually know if what they’ve written makes sense legally speaking right but the large part of it could be automated and then after the votes these people look at their law written in the in a Word document and they manually apply the Amendments of the bill and then the bills below and so you have these highly qualified highly paid lawyers paid to do like well certainly interesting relevance legal work but also manual Version Control there’s no CI CD as well in parliaments and we proved out that imagine for example for the taxes we have a budget every year this is a perfect test case for taxes how much are we going to get in tax money how much are we going to spend if I change the percentages on something in some law what impact will it have so all that kind of stuff this could basically also be completely automated but we’re not doing that but anyway so that’s not so quite far from

[16:01] Developments but I was just to say well sometimes the frontier between what can be estimated what cannot be is strange and interesting and that’s that’s always a an area of Wonder I think in our job it’s like how far can we go in Automation and can we solve the problem by just modeling it better or do we actually need to run big machines on huge test sets I think it goes without saying that a lot of stuff can be automated in general in the world just the fact that we are software engineers and we tend to have some experience with automation means that we are right there at the right place to automate it but of course if one of us has the ambition to transition into French law we can maybe fly the automations there help the French Parliament into the future if you’re not using our Parliament anyway these days because the executive passes the bills without going through a vote so that’s a perfect test case for people like us right right you mentioned some interesting things Pierre Chen and it made me think about using distributed Version Control right now and I think

[17:11] Most of us are using git but you can elaborate on that when I ask you the question later the human factor is also a factor when you use a virtual control system right I have seen a lot of co-workers struggling with their git repositories and that was I was wondering ishel maybe what version control system do you actually prefer and what have you seen out there in the wild when your co-workers use Version Control that they also kind of wasted some of their time because they ran into some problems there in terms of tools what I have seen out there actually I started with subversion at the beginning of my career I actually had to deal with it I have a brief encounter with Mercurial in some of my customers but most of them and since some years now most of the people that I encounter developers or companies that I collaborate with it was mostly git cute it’s an improvement into what we had before I mean having complete code based repeated different centralized ways of sharing the source code was a little bit complicated and then for example having to resolve all the conflicts before making a commit

[18:35] Instead of git like committing your actual work and then trying to figure out the conflicts it’s it’s also an improvement how do you see your work and how do you see yourself collaborating with source code a large source code the challenges are obvious I mean at least for me it’s our code base are increasing in size in such High speed Pace that we need actually tools that are easy to use and can keep up with all this code bases extremely large code bases and with different kind of binaries or files kit it is very efficient for some things but also allows people to create to get really frustrated as you mentioned Hano I have seen and I have people referred to kid and how do you get from point A to point B and saying well you can get from point A to point B like a snowflake totally different ways to do it and I have seen people that use the ref law like their best friends and I see people that don’t even understand what the rough lock is I see people going and Driven Crazy because of all the merge conflicts and when they are seeing the merge conflict and says okay three commits solve

[20:14] 180 to come and they’re like oh my God what should I do I have seen bad practices rewriting history I have seen so complicated workflows that you are like what were we doing in the first place like I mean we’ve been working on trying to fix the state of the files but I at this point I even forgot what we’re trying to achieve here right now I think git is the most popular one it’s very useful it’s very powerful I think it’s the industry almost default I think you still can misuse the tool and I think there’s still a lot of unknowns there are still a lot of best practices out there but I still think that the level of usage of git in the normal developer is not proficient so I still think that we have UI interfaces that obscure a lot of the main knowledge of git and as a developer in my perspective and in my way of Behaving I like the command line so that gives me a lot of work of control I know there is I know but at the beginning it was the best way to actually tame down that Beast anyway I think I hope I answered your question Han yeah you did actually this really brings a few familiar bells for me so at

[21:52] Info support I teach a few courses and one of them is good for developers it’s one of the first courses that are the people that start out at our company do and I always try to teach them to use the command line but also one of the first things I show them is a wonderful website that is called dangitgit.com we can add it to the show notes it’s a it’s a very funny website created by Katie Seiler Miller dang itgit.com and it’s just a knowledge base of stuff you can do to fix up your repository when you did something wrong because doing something wrong with Git is quite easy but then fixing it well that is just a bit harder and it’s just a knowledge base for Katie to be able to refer to it back again when she encounters a similar issue so I completely agree and recognize the Things She Said Trisha what are your experiences here I mean do you like the git command line I got the feeling from Excel that you don’t really but you can elaborate I mean git is my default as well I mean everyone uses git these days I actually agree with a lot of what HL says I don’t mean this is a criticism of

[23:02] Normal developers of like the majority of developers when I say many developers don’t know how to use git properly it’s not their fault it’s that the tool is so powerful so huge it has so many different variations come is used differently by almost every team in the whole wide world it is impossible to know the whole tool it’s impossible to know how to use it properly and one of the things I want a slight slight plug for my own book one of the things I tried to teach in the IntelliJ IDEA book I wrote last year and this comes back down to the command line versus the UI so IntelliJ IDEA has a really nice UI for working with Version Control I’m not really a command line sort of person so I do prefer using the UI when I was trying to teach I indirectly teach Version Control while teaching it through the medium of IntelliJ IDEA the idea is just to try and expose developers to the smallest amount of stuff they will almost definitely need to use for example committing pushing branching merging what’s emerge versus a rebase I don’t know if I added cherry picking I think I might have decided that let’s not cover

[24:08] That there’s a lot of information about get out there there’s a few great new books out there as well including head first git and another book whose name can’t remember but I’ll try and give it to you so it can be in the show notes a lot of great new resources on git but for the longest time a lot of people were trying to teach git from the command line in terms of teaching like here are all the features pick and choose the ones you want like cherry pick the things that you want and I think what we really needed to talk developers is like here is your entry level git here is the kind of like level up and then from there this team uses git in this way and these are the things that you will use and this is how you will use it and I think that way we can get a bit less confused over what’s Reba rebase versus emerge and when do I use these various features I think that it just becomes very complicated to understand these things however I’m also a big fan of Chunk based development and I’m I’m not a big fan of long lived branches I don’t like the idea of having a long-lived feature

[25:05] Branch which needs merging or rebase and rebasing along with feature branches like the most difficult thing that you can do you need to do regular merging between various branches I don’t like the idea of having a branch of work which is you might be running your tests on it but you’re not doing continuous integration because continuous integration means integrating it not just running your tests on it so these are some of the other patterns that you can have in git that I personally don’t like because I think it makes it pushes a lot of the a lot of the pain to the end of the development process by which you get to the point like your child was saying like by the time you start merging or rebasing or trying to actually integrate your work you’ve forgotten what you were working on the code base has moved on from where you started you’re not really sure how your changes are impacted by someone else’s changes and I have often seen people going I don’t know what this diff means I’m just going to take my changes and trample all over someone else’s or I’ll take all their changes and just rewrite

[26:04] The whole thing and this is when we get back to manual processes and Toil and pain I think that some of these tools just make can make it too easy to do things which are make developers lives difficult but again it’s all about the people it’s all about the team it’s how the team chooses to work and it’s about having a consistent process across all the people in that one team and understanding how things are done and also regularly integrating in my personal opinion very familiar of things that you brought up there so Pierre Chen I was wondering of course I know you’re from the people virtual control system and I was wondering if the things that Trisha just taught us also inspired you a bit to start working on that long-lived feature branches and conflicts that are very hard to merge literally every single sentence said by icha and Trisha in the last five minutes were big triggers for me I’ve had to take notes actually so there’s a lot for me to comment on I I’ll try to be brief but it will be hard well I actually said something that I found really very relevant she said that’s

[27:12] A few developers are proficient in this so how is this even possible that the two that’s 18 years old that’s widely used by everybody in the world is in the state where a few people are proficient in it I’ll actually elaborate on that and go a bit further how is it possible that nobody outside our industry uses gives it’s it’s supposed to be one of the great Prides of computer developers that we’ve been able to build a tool as that is this powerful and useful in relevant to all kinds of projects well except not in the actual world so it’s relevant to all projects but it has to be it has to follow good practices so good good bad practices were completed many times by both teacher and Trisha so yeah good practices so what’s a good practice let’s say I want to build something with my legal search for example like I love playing like all right so I want to build a I want to build like let’s say a car in Lego so what’s a good practice for that there’s no real good practice like we should just but like play and let the tools do what they’re meant to do like if I want to use screwdriver on

[28:24] A screwed so what’s a good practice for that well just use it so it’s it’s striking to me that we’ve not been able to export our Technologies to outside on a little tiny field like while computers computer science is exactly a tiny show but anyway for example these days who doesn’t use stuff that was invented at Nasa for example everybody uses NASA invented stuff like from silverware to outdoor equipments to like anything really like in NASA is really proud of it and communicates heavily with that who uses stuff not in comp in Computing that’s that is invented at kids nobody I think that’s striking and it’s it’s not because the tool is too young tool is like 18 years old 500 different commands like sub commands and that’s something I think I remember I think Trisha said it’s like there’s lots of lots and lots of commands and nobody really uses it like knows every every single commands one link I’ve posted in this chat is the gitman page generator it’s a it’s a website that generates a new man page for kids randomly every time you click on the main pages are obviously completely fake

[29:42] But they sound real like when you read them you’re like oh yeah this could definitely be a good command this is incredibly complicated so yeah what inspired us to start people was that we needed a tool that would be that would be that would allow us to do everything git can do but also be simple and while I mean simple to use really fast that was a requirement and I would let people just work instead of teaching them how to like teaching them the good practices telling them whether to merge to rebase Cherry big well all these three commands in people are the same one you just apply a patch and that’s actually what you want to do when you when you’re rebasing stuff you’re just using past commutativity to swap the order for example so and that’s the default in people you’re you’re it’s the default because you don’t need to swap the over the order is irrelevant if you can apply the patches in two different orders you don’t need to swap the order because it’s the same and that thing even if you don’t swap if you know if people if you don’t use if you don’t think you’re

[30:51] Using patch Community you’re still using it you’re still benefiting from us so just some context here so people with a tool that’s based entirely on patches instead of snapshots so commits or snapshots and get rarely uses patches and in people we do the offices we focus primarily on patches and the user only sees patches and then behind the electorally or the group we’re we’re using we’re using a giant data structure a graph like data structure that’s used to merge all the batches so the batches are not exactly huge expansions they have a little bit more information but we’re using a data structure that’s actually a crdt that wasn’t found in this way it was found by asking the question it was a purely theoretical question we want to we want to handle changes files right so what is this so okay we can have conflicts sometimes and conflicts are bad and nobody likes them right so what is the question the initial question was what is the minimal extension into files that can tolerate confidence and what does it mean to generate conflict so this asking this

[32:05] Question means what is like you can merge you can merge or changes you can apply your patches all the time always sometimes you will have conflicts and because you’ve you’ve asked that question mathematically you know that your data structure will model countries properly so you can apply your patches you will have conflicts and at the end of sequence patches you could live you can ask the question okay what are where are my confidence and then you can solve them you solve them by creating a guess was a badge and when you send that patch out to others we may have the same conflicts but it solves them for them too so you don’t have to worry about whether my branch is long lead short-lived or feature Branch we are just using branches and batches and that’s it’s much simpler you have way fewer Concepts and you can do the exact same things well except for producing the bad merges it gets which you can very easy to get even even by following good practices yes this these all these issues are the two reviews have mentioned were definitely a starting point for us yeah thanks PR

[33:14] Chen so quick follow-up question here I’m assuming you’re useful yourself yeah I don’t think I don’t use any other version control system yeah yeah I’m not surprised you said you can either choose to use short-lived branches or long lived branches with behold that it would be just as easy so what is your preference there well it depends I don’t use I don’t use branches too often to be honest the my favorites model of branching is patches the reason we use short clip branches or feature branches or trunk-based developments is that we want our different features as we dependent from each other and why do we want to do that because we want to be able to maybe push them independently from each other or maybe undoing them right and when you when you have a system that handles patches properly this is something you can do even without damages so for example in people suppose let’s say you have two different patches that introduce features and did what you would call them branches and that would it would require you to plan your work ahead of time say

[34:24] Well now I’m working on this feature that’s not all the way I work well maybe it’s not suited for everything because that’s not all the way I usually when I start work in future all things are broken always and I discover a bug which leads me to believe that I shouldn’t be working on another feature instead so that you like in YouTube adapts you use all the Arsenal of stationing and staging and whatnot right within people you can just write patches well this is the only command we have so you’re you’re bound to use it anyway so creating a patch applying a match that’s all we do and so yeah my favorite model of branching is fascist so you’re working on two features in parallel just make two batches and then you’ll be able to pack to push down independently and you’ll be able to unrecord undo them and depend it if they are not if they’re separable actually so if they are if they depend on each other they touch the same place in the file or then you will have to be a bit more rigorous and just like meaning to just do it and then solve the conflicts afterwards if you

[35:37] Want back to do it to do the actual branches we do have branches but you can also use batches and so another model of branching which I’ve also used sometimes is you want to move you want to maintain two different versions in parallel like let’s say for example the feature in the language you’re using is not quite ready but you want to still experiment with it and all the bug fixes your you find while experimenting with that you want to back prop them into your branch so there you’ll have two Long Live branches and they’re necessary for your users for your customers because your customers want to be able to like two two different classes of customers want two different versions and then you’ll be able to just backboard you won’t have to backboard you just have to take the bug fixes take the bug fixing patches and just apply them to your previous to your previous branch and that’s it and then you when you merge everything later because the patch when you apply them don’t change their identity you’ll you’ll still be able to merge all the other changes and you won’t get any

[36:41] Merge conflicts because the patches are the same like the battery one single patch is the same no matter where it lives it can be applied to different branches in different contexts that won’t change its identity so you will still be able to merge the very the other yeah they’re like the new experimental Branch without without losing your mind on you know conflict resolution and all that kind of stuff Trisha you also mentioned trunk best development just as Pierre Chen did a few moments ago one thing we could do is switch typical I’m all for it but still have to convince my co-workers and the client side then are working but I think a few of the trunk-based development ideas can still be applied regardless of what virtual control system you’re using right I read a blog post on a couple of weeks ago when I worked with Dave Farley who wrote The Continuous delivery book we were doing well to be fair we’re using subversion but even if we haven’t been using subversion we would have been using trunk-based development the idea being that everyone

[37:43] Is working against chunk they don’t have their own branches they’re not they’re not doing feature branches the only Branch you have is that the code that’s on your machine is separate to the main to the main trunk and it works in git as well you make some changes you make sure everything’s compiling you make sure all the tests are running you commit it to trunk to the main branch and in that way because your CI is working all the time on your trunk you know that well your acceptance test pass all of your performance tests meet the demand demands and your trunk is always ready for deployment as well because this is an important thing for continuous deployment the idea being that any point in time you take what’s on trunk you know all the test paths and you just deploy it whenever you want to when I was working this way with Dave Farley it was a few years ago and we were doing like releases every few weeks and I know nowadays lots of people are doing micro services and they’re releasing potentially multiple times a day but to me that doesn’t mean that you

[38:43] Can’t still do a development model where you are pushing your code into trunk all the time if your feature’s not ready it should still be compiling even if it takes you two weeks to create a feature because for example I was working on a financial exchange and some of these features take quite a long time to implement new instruments or new ways of working with the account but you can be using feature toggles or literally just code paths that can’t be reached until you actually connect them up via a button on the UI or whatever it is but all the time that you’re working on this feature even if it’s a long-lived feature you’re still working on the main code base you’re still not merging the main code base into your code base the only way you’re merging it is when you do a get a Fetch and you know that your code is always compiling and you know that your code will continue to compile against against trunk because you’re constantly working against trunk you’re also getting all of the other developers fixes and changes as they work against Trump too now I did write a blog post about this

[39:41] Recently I understand there are like with anything there are lots of it depends and criticisms and places where it doesn’t work so for example if you’ve got 300 developers on working on a huge application which is all in one module in on trunk maybe maybe you’re going to end up with a lot of problems and you’re going to be triggering and triggering an awful lot of CI Builds on the same on the same branch on the trunk however I think if you’ve got 300 developers working on one large mono repo I think you’ve got other problems as well I think the movement towards microservices came because now we have a lot of big code we have you know like I think Michelle mentioned this as well these code code bases are getting bigger you’ve got more developers working on the code one of the ways to address that is to split it up into separate services or separate modules separate teams so no of course something like trunk-based development doesn’t necessarily scale if you’ve got 300 Developers but if you’ve got 300 developers you have a scaling problem anyway probably so yeah so my preference is that the

[40:45] The core of what you’re working on be it a micro service or a module or a project all the developers who are working on that should be committing to that main trunk the only real exception is something that Pierre Tien mentioned which is that sometimes you might have the branch that’s in production and the branch that’s not in production so I think that’s the that’s the exception there so Trisha what would be your advice for people who are currently in a feature-based approach but really want to transition to trunk-based development and also maybe second question maybe you can address it as well what if the context you’re working in on a platform level demands that you use feature branches because it’s linked to tickets ticketing system IDs or something to give us some advice on that I’ve been thinking about this a bunch and we have to bear in mind that I’ve been doing development advocacy for the last like 10 years or so maybe I’m a little bit divorced from the reality of real Developers having said that the flip side is I do quite a lot of Open Source development which means that I often wander off for

[41:49] Weeks at a time don’t do any code and then come back to an open source project and of course it’s moved on since since I did some work and one of the most important things in these environments is to continuously pull from what’s new into your branches into your fork and I think this is one of the things that I have seen go go wrong in long-lived feature branches is that people start a feature branch and I’ve worked in teams like this you start a feature Branch when you start that feature and like you said you’ve got a tracking number for the issue number or the bug or whatever and you start working on it today and you finish it next Thursday and during those two weeks you haven’t pulled from Maine at all and that’s when you haven’t merged anything any of the other developers stuff into your branch and that’s when you start getting like merge hell because you emerge it you may hopefully merge it at some point before you do your pull request or your merge or your rebase if you’re regularly fetching from what’s in trunk then you end up having smaller lots of changes that you need to pull into your code you

[42:53] Have fewer chances of merge conflict and you’re working on the most up-to-date code which means that as soon as someone has worked on something which is going to conflict with yours you see it sooner rather than later so one thing is I would definitely pull more regularly from bromate Trump if you have to have long live feature branches and the other thing is to try not to have longer Fitch branches it’s really important and this is a computer science problem since forever this is not a good thing it’s important to think about things in much smaller pieces of work than we generally have been right you shouldn’t be working in isolation on one thing for two weeks a lot of people asking me after the blog post what’s a Long Live feature Branch anything longer than 24 hours really because you can’t be working in isolation for that amount of time you need to be able to split stuff up if you’re working on something super separate it’s kind of easier in a way you created a new instrument class let’s say for the particular instrument that you’re creating you can commit that and you merge it into Main and nobody’s using it

[43:51] So it doesn’t really matter there’s no impact to that so small changes very small changes and if there’s no conflicts and nothing in the way merge it back into trunk as soon as possible if you can’t merge it back into trunk for whatever reason just continuously pull trunk back into your feature branch so HL you’ve heard the Pierre Chen talk about his version control system for a few minutes did you hear some things that you where you thought about and thought I would really like to have that in my version control setup or maybe you can talk about a few other improvements that you would like from your version control system you must have been thinking about this in the past years I mean I think every get user has I like it but it just needs to add this extra feature about mentioning it’s really interesting because the workflow that he described it was very very easy like a communication between humans and saying like this is what has changed and because you don’t have so many open contexts then it’s easier to keep track to what is actually the change because let’s face it as humans we can

[45:03] Only retain very small amount of things in our head in our work in memory so going beyond that doesn’t make sense I mean we can be working in 10 different features but not at the same time we can’t like that is not human possible so at some point it’s what Trish is say like you have to reduce it to the very minimum work on it and serialize and I sadly for all the synchronous that we like to talk we are very serial like humans we focus on this and then on that we can have different ideas during the day and that can be totally asynchronous and in parallel but working working is something that we have to start and finish at some point so the Simplicity of our tools that allow us to actually focus in a specific thing makes total sense for me now the other thing that brings a lot of problems into development is the human factor and I put a link about conventional commits because my whole point is I’m not going to even talk about Long Live feature branches from no it’s like write down what not what why you have introduced those changes what you have the freaking diff there that’s not

[46:29] A question I can see what happened but not why a pattern I agree with your team what makes more sense the whole history of kid should give you a more enriching idea of what it was happening what where what actually maybe not what we were thinking and we didn’t go in that direction but at least hinting out like this is why it actually happened not what actually happened but why it happened I think for tools that follow better the human interaction of how human works it’s going to reduce the overhead for us using it and we will be more efficient using them so that’s why it totally makes sense for me please be kind to your fellow developer because you may be the one that is going to go like what were we thinking like wait were we thinking at this point I really want to follow up on that as well the why not what on Commit messages particularly again coming back to the IntelliJ IDEA I quite often use the git annotate on the IDE so when I’m looking at a line of code and I see I don’t know some weird if statement if I use annotate and it says and it has the commit message there in the IDE and it says I was working on

[47:55] Something which makes sure that users don’t get screwed over by something something you’re like oh right now I know why this if statement exists so now I know what this test is trying to do whereas if it says added a test to make sure issue number 521 works I’m like that’s that’s not very helpful so all what you’re saying is which we should be talking about changes and you are actually when you’re talking about comments we’re talking about changes yet you’re using a system that only stores snapshots so git is entirely about the what the snapshots was an entire version of your project so you’re entirely about what is there you’re not you’re not talking about what changed since last time in order to get that vacuum gate you have to recompute it from two different comments and you you’ll have an automated machine generated answer that doesn’t that might make sense but sometimes doesn’t like I have examples of where of like giant git figures they’re obviously not too common but merges where Alice and gloves are working in parallel and this is working towards the beginning of time

[49:02] And Bob is working towards Diana right and when they merge Bob’s new changes so identified are inserted into Alice’s new changes and that’s a common that’s a that’s an issue in three-way urgent there there’s just not enough information exactly to work so you could not you could not possibly solve the merge problem that git is trying to solve correctly in all cases this is just to be necked out you’re talking about what you want is changes and what everybody wants is changes nobody’s ever seen a comment actually except maybe git developers what when you when you are trying to visualize the companies in any UI you get presented with a change and that’s not what the comment is and you’ve probably since you you’ve died you can see for all these years you’ve probably seen countless examples of junior or maybe not so Junior devs saying what no no let’s say possible the comments is a change no comment is not a change it’s not a diff I think I already encountered some love for some of the features of pickle with the other two guests that we have and I think it would really fit well with this trunk based

[50:14] Development approach that where that we have been talking about so PR Chen I’ve been meaning to ask you do you understand why so many people are still using git currently can you think of a reason why they are I can’t think of many reasons one thing I haven’t stated since the beginning of this podcast is that I’m a big good fan actually I also use the command line and I also look at the reflog and I love how it’s it’s just like it’s just not meant to merge because it’s it’s it was designed by people who don’t like like they have very strong opinions on how they collaborate with others right currently or kernel developers it’s a very exclusive team working in one very rigid way that is now suitable for all kinds of projects it’s also a good Linux kernel is its own thing right it’s it’s probably the most the farthest outlier in all software development projects that there is so and we’re all taking taking their developing tools and saying well if it fits the Linux kernel it probably fits our project but no that’s that’s completely wrong yeah so one thing is yeah it just works

[51:27] Git really really works and it works well it’s very elegant so I used to say simple because I take that as compliments but some but sometimes sometimes people say look at well they see simple and they’re like oh it’s simple there’s probably this probably not the future but like actually being simple is actually a really strong feature designing simple systems that work is great and so you can get it simple now I prefer to stay elegant because it’s it’s less ambiguous let’s say so yeah this is really I think it’s data structures are already really small and they don’t do much right they don’t do they are rebates as well they’re going to their merge as well they don’t do they are completely unable to charity even though they try to really make all these commands they’re unable to actually Implement them fully properly rigorously close yeah that’s a great tool so it works it actually works people is much more complicated it’s very simple to use very easy to use but it’s also extremely complicated under the hood to implement it all like to make it fast like for example I’ve

[52:33] Had to write database back end especially for people so this was a nightmare price like it’s called Santa Kevin’s dictionary in Finnish it’s just a key value store which turned out to be the fastest open source key Value Store there is but it was also a nightmare to write and the reason I needed to write it was not even performance the reason was that I want to be able to Fork to form a key value table so when you have a dictionary you want to be able to just Fork it and get two separate copies without copying a single bias so that’s what Santa CLIA does using persistent data structures garbage collection reference counting and so on so but this is a time like a giant machine it’s really it’s really complicated like there’s there’s few lines of code but it’s good it’s not easy that’s one reason but I doubt this is the main reason because very few people know the internals of this I think one of the reasons is that well it works people have nightmarish Memoirs of switching tickets when they switched from SBN with all all these little hacks that they were using and

[53:45] And they switch to get I know a trivia from some companies where people actually physically fought over this issue of switching to this like during the meeting two men obviously women wouldn’t do that I guess so two men at some point say well let’s settle it there it’s just outside the building so now they are looking at a new two with people or whatever and they’re like no we’re not going through that again so that’s that’s one of the one of the issues another issue and it’s it’s actually appeared in the chat in our Podcast chat here the tools aren’t quite ready yet for example for git we have GitHub it’s it’s it hasn’t innovated in any way in a number of years but it works there’s 500 developers taking care of your Source codes for people that we we’ve had some well like I wrote something called The Nest I released it with the version of people back in November 2020 I guess it’s it’s okay but it’s with it went through a number of issues like for example the ovh data center fire in March 2021 while my server was in the was in the fire so I’ve had to rewrite a

[55:01] Replication layer to be able to replicate it in three different data centers so I did that suppose the tools that this uses like password SQL don’t really work well because I want to do like the main issue in summary one was that I wanted to do a join like a database join between information that’s stored in beautiful repositories and information that stored in most Grace gear so in possible experience you can easily do a drawing between two tables but you can option to do a join between two database databases especially if one of them is not even strong breast because it’s stored in the form of people requests so this doesn’t work very well and it causes the machines to go down sometimes not that actually go down but just be under too much load and so this should be okay but actually because because of their of how fast price QR works it just sometimes gives its data so that’s really bad having downtime is okay losing people’s data is not so I’ve decided to rewrite this and released it recently but we’re we’re currently going on under a major with rides and redesigned the ecosystem

[56:12] Around like repository hosting so you know for example just saying this these things this problem is probably very scary for lots of teams so vehicle itself works and works very well and you can use this randomly and repositories or very unlikely to encounter bugs we’re testing it all the time on crazy scenarios and huge repulsaries and so on so the tool itself is ready but the two moving around it like hosting hosting services and so on we’re currently finding a better way to do it it’s probably just a matter of weeks or something because it’s the you know the majority of the system of the ecosystems yeah because you mentioned a few possible reasons why people would still be using git so I wanted to ask you two small questions the first one is it actually one of your goals to grow people’s market share and secondly if yes what do you think would be needed to achieve that growth well the initial goal was that problem was interesting and that’s how phloem Baker and myself started working it wasn’t we didn’t have any idea that in the results and two like that or like okay docs has

[57:29] Huge performance problems and flowers very knowledgeable about darks because it was one of the maintainers and so he told me well no one really knows why the merge can sometimes take exponential time and we’re like oh actually we’re computer scientists working on that all the time on asynchronous Computing models so this is a this is a job for us okay let’s do it and so we looked at the bibliography the installed noticing some some cool algorithms that we could use and we needed to expand on them and so on so this they started because the problem was interesting but now yeah growing marker chair you know I think our industry is quite conservative in that regard it’s it’s hard to make people change we like to see ourselves as the heroes of progress and modernity actually were very very conservative when it comes to our own methods this will be this would be a growing growing people’s market share in the in like in the currents markets of kids will be hard because there’s another Market that’s completely unreachable I guess that’s people not not even doing code legal

[58:45] Documents office documents all these people they desperately need Version Control and they cannot possibly use tools like this even if you restricted their files to formats we know how to handle a git like markdown later and these things even that would be really hard to use Google go teach a parliament member how to rebate and whether they should merge or rebase or interpreted something from another level this is completely possible so same thing for lawyers Liars worked really fast faces like most much faster than that and so they thought they could not possibly spend half a day on dealing with two branches that have gone wild because one of the junior junior Liars in the team decided to do a reverse these markets are definitely addressable by equal and there’s another there’s another niche in our industry which is a video game developers so these people are don’t see their needs addressed by anybody their most most teams are currently using proforce which is a svn.lex the SVN was CVS dot right and Perth versus espn.com so yeah most most video gaming teams are still

[1:00:08] Stuck with that sorry I really wanted to follow up on something you said there which is absolutely fascinating to me it’s difficult to it’s impossible to persuade a lawyer to spend half a day like fixing up a merge which has gone wrong and this is something which is this comes back down to the DPE stuff that we’re talking about a Gradle as in the software industry we’re kind of fine for really expensive software Engineers to spend half a day or a day mucking around with mergers and these are the sorts of things that get in the way of our productivity and yet we kind of tolerate them it’s fine to spend a thousand dollars on the fact the merge went wrong and I think that we have become very tolerant of this kind of thing precisely because as we don’t want the pain of changing tools or changing or migrating and I get that too because migration is an enormous mess but we will tolerate small pains like half a day here and a day there on tools and processes which are not working for us and I think that this particularly in the development environment in the production environment we’ve got a lot better at

[1:01:10] Back circling back to where we started this conversation got a lot better at automation pipelines smoothing things out but when it comes to the development environment we still tolerate this nonsense of expensive Toil and I think that’s kind of crazy yeah I think Liars will just laugh at us yeah it seems to me that before could also really benefit then from plugins for popular Ides or maybe graphical ways to show the state of your version control because lawyers are not going to learn the whole command line I can imagine absolutely yes and it’s very the people model is very suitable for that you can you can easily visualize patches and imagine is something everybody understands and you can also have fetches that are not necessarily on lines but they can work on words for example or any unit actually can change the if algorithm as you as interesting so Excel I wanted to ask you a question that is about 10 years into the future suppose we could fast forward 10 years into the future right now and we would be able to compare that state of source control and build automation to the

[1:02:16] Current state that we have right now what do you think would be the biggest difference certainly as we have heard even though we have as Pierre mentioned a very very elegant tool right now I for the comment that we have set this is really not the best suitable like 18 years not all developers know how to use it so it’s full potential which either points towards we are difficult to teach or maybe the tool is not working it’s like when people say to go against the grain maybe this is the case so in my point of view there are so many things that we have learned from managing a large amount of information that changes in a strategic or in a specific way is this the best we can do certainly not the numbers speak for themselves the experience of the developers speak for themselves so I know that it has to be a change one of the changes that I have seen but it’s a totally different thing it’s you have your kid and you’re building a lot of other tools on top of it block tracking cicd documentation etc etc so I have heard that and I have seen that a lot of people when they refer to the next set of tools they

[1:03:47] Refer to those more things to do near the source code but that doesn’t mean that it’s a different version control system you see the difference so I think both are going to evolve naturally we will have to find a way to actually make it a more interesting for developers to use efficiently proficiently and easy something that is really important I mean we’re talking about people collaborating in large code bases at the asynchronous Times and we cannot only relay your best practices because best practices are amazing but it’s not a rule and we are not very good at not following rules so the future something different that I hope more developers and even non-developers can use without going into the merge hell that we have spoken at so many points during this conversation I think a lot of other tools will leave very very close to Source because we have seen it we need cicd we need documentation we need even I think testing will because it’s so important it will leave in a different way right now being part of the source code is super important it’s really important but now we have

[1:05:22] More Integrations and suddenly you start seeing like developers don’t know how where to pull them or like to put them I for example consume an integration test consumer from the producer perspective you already have problems like locating it in some way so I think it will actually live in its own close but in a different way that’s what I hope something that works efficient for the computer the computers can work with it in the most automatic way but also developers can work in the human way too and Trisha what do you think 10 years from now the world of source control and build automation I think Michelle’s right I hadn’t even realized it was right for change until we had this conversation but it’s it’s right for change I think there’s a number of things that we’ve talked about today which are really important for example we’ve seen that the git is very fully featured simple ish like it’s implemented in a simple way but unfortunately exposes itself in a complicated way so it means that developers have to know a whole bunch of different ways to work with it and then figure out their own human best

[1:06:36] Practices to layer over the top or like a child was saying that maybe having tooling on top to make it simpler to work I really like what piitian said about what if you know you have something which is simple to work with and complicated underneath I mean that feels a little bit like Java the programming language right from a programming language perspective it hasn’t evolved that much but it’s kind of simple to understand but the JVM does a lot of hard work for you and you don’t have to worry about it that’s why it’s one of the fastest languages in the world and you don’t see that and you don’t have to code for that and that’s where I would like to see our Version Control Systems go so you use it just works and maybe it can evolve in the background without you having to worry about new commands new ways to work with their new best practices so I think something which is a fundamental building block that we don’t have to care about helps develop a productivity because you can automate that stuff and not worry about it and then we can worry about the things we care about I think layering on

[1:07:34] Process and people stuff over the top of something to fix the holes in our tooling is not sustainable it’s not scalable we can’t keep patching around these problems by having for example code reviews and pull requests I’ve seen a lot of complaints about how what if you have a process that doesn’t have code reviews or pull requests how will you be able to function as a team well what if we have things like pair programming which I prefer or what if we actually invested in our Juniors so we could rely on them to write code instead of assuming they write terrible code and having to do tedious code reviews at the end of it these kinds of things what I really liked about what prtn said as well is like this idea of as as developers in particular we like to solve problems so we see a problem and we solve it by perhaps layering something over the top of it but what if we could step back and go what if we can eliminate this problem what if this problem is a side effect of what if it’s a smell of an incorrect tool of something which can now be fixed by a different way forward so I think that

[1:08:40] Papering over these problems with processes or extra tooling it might be better to step back and figure out actually what should we be doing instead the other thing I wanted to say is we haven’t mentioned AI at all during this conversation right and in 10 years time I think AI is going to be a much larger part of the developer tools set I don’t know what it looks like yet I don’t think it’s pairing with chat EBT but you know who knows I want to kind of circle back to this idea of let’s not use AI to paper over problems for example let’s not just use it to go well maybe AI can do our code reviews maybe I AI can do our merges okay but what if we didn’t need to do those things one of those are better faster easier cleaner way so we didn’t have to keep layering on more tools and more Technologies so I think that’s the only thing I really really want to say about AI I think it will probably contribute to this space I would like us to not just add it as an extra piece of noise for us to debug later on down the line Trisha I was so looking forward to talking with you for a morning without having to

[1:09:42] Mention AI once no I don’t blame you at all of course that’s a major influence and I tend to agree with you Pierre Chen you have the privilege to utter the final words of our conversation what do you think of the world of Bill Automation and Source control in 10 years anything you would like to add lots of things and nothing at the same time you know as Yogi Bera said predictions are hard especially about the future in this case I think we’ve seen an accident like a dramatic acceleration in the changing like in the pace of change that and again Technologies were used for example if you think about not 10 years from now but 10 years ago Docker was just beginning there was no go Scala was traveling like we were talking with Java developers here so oscara was a thing but not as widely used as today there was no like mix OS was absolutely unusable and there is no you know angular reacts in the JavaScript world there was no like Web 2.0 was a thing but not actually a thing so and we did have tensorflow for example for machine learning if you look at all

[1:11:01] These and I’m probably forgetting lots of stuff right if you look at all like yeah rust rust was not it was a prototype from Mozilla aiming as a like slightly better plus boss yeah predictions like things are things I’ve been accelerating dramatically in the last like 10 to five years and so I don’t know what the future will hold that is it’s really hard to tell if I were to give this like a tiny prediction about about Version Control only because it’s a start topic I know where it’s quite quite of this yeah I think people will probably maybe maybe some teams will switch to something like like or maybe some maybe not payroll itself but maybe like the workflows were allowing people to do I think this will be this will be used elsewhere we’re already seeing seeing the influence of the things we’ve been doing in other products like like even at GitHub and gitlab like people are using like the experimental things we’ve we’ve been we’ve allowed ourselves to do they’ve they’ve been used in you know in other products like for example

[1:12:21] Sign in by defaults and in GitHub that’s something that’s we did first and well discussions instead of full requests so people as a merger between discussions like in the nests we’re using a completely different model from Full requests we’re using just discussions and you can attach passions discussions there are not full requests or merge requests they’re just yeah you can start discussing on the topic and then push it back and then say well I actually have a solution to that like we should patch the discussion then you can review the patch independent of its context and just apply it as we want to your main branch on your computer because you can always outdo it you can apply it to different contexts and so that you can test it like that so we’ve all we’re already already seeing the influence of the stuff we’ve we’ve done in other products GitHub has introduced discussions signed commits by by default this kind of this kind of thing so yeah I don’t know the future we’ll see that’s always the best answer I think we can all agree that it’s a

[1:13:34] Very interesting field that is changing rapidly and I’m planning to be there every step of the way because I really like where the field is heading I had a lot of fun thanks special thanks to our guests Trisha it’s shell and Pierre Chen for sharing your thoughts on Version Control on build Automation and of course we want to thank you for listening to this podcast keep an eye on Foojay for future articles about development and everything related to the Java world thank you and until next time