Foojay Podcast #74: JCON Report, Part 3 – AI, ChatGPT, LLM, ML, RAG, MCP, GenAI, and more

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] Let’s have an AI bingo and talk about chat GPT, NLM, ML, Rack, MCP, GI, and more. Welcome to the Foojay podcast. All your news about OpenJDK. This is part three of the interviews recorded at the JCON conference in May. In the previous part, you learned more about how you can be a better Java developer and how Java has evolved and keeps evolving. But of course, artificial intelligence and large language models were a hot topic at the conference. I collected all the interviews around this topic in this episode and you will learn more about the different technologies we can use in our Java projects. I also checked with my guests how to compare Java to Python for AI related development. Let’s start with Pasha Finkelstein who had two talks at the conference. One of them being about converting from a jet GPT user to a pipeline creator to process documents.

[0:58] I asked him to explain the difference between rack and MCP to hot topics within AI. They’re very different. I think that MCP is essentially a protocol for AI to communicate with some sources or some tools probably find information there, but it’s much much wider. can tell. We actually have a booth here right here that says with MCP allows car to drive. You type a comment into a chart and the car goes forward, turns left and so on. Very very simple but nice to rack is all about prompt augmentation like we need to find a relevant information to what the user actually asked. put this information inside the prompt and ask LLM what it sees inside this information. For example, we just had made this experiment with K Horseman. There is a new that about structured concurrency in Java. It’s 502 if I remember correctly. So we loaded this chap inside our vector database my machine and asked chat GPT to give us an example of code using the what it learned from this from the from this jab and actually the answer was very good.

[2:20] No hallucinations no anything and the new syntax that was introduced in this jab. So summarize track is to help the model to know more about your product for instance. yes mix some data some actual data into a prompt to help model to answer a questions in a more relevant way not just hallucinate or not search somewhere inside it’s I don’t know black inside it black box rights and measures we give this information in plain text automatically and model like okay I know the answer or if we didn’t find anything relevant then model will say sorry I don’t know anything okay is Java good language to build this kind of things because yeah a lot of people think Python is the way to go absolutely python is a way to go when we want to build AI itself because all the tools are there PyTorch Kas and so on but if we need only to communicate with tools Java is amazing actually there are two technologies spring AI and LangChain4j J which give you certain level of abstraction from all these models and so on and they actually solve problems for you like how to communicate with the model how to put data in vector database

[3:45] How to create a prompt everything almost everything there is automated essentially the demo I gave in on my first talk was like 20 lines of code or so and it was very very simple my code was in cotlin but it doesn’t make any difference I just wanted to use cater HTTP client. That’s why I used Cotlin. Otherwise, I could as well use Java. Isn’t it amazing how fast something like LangChain4j and Spring AI was developed by the community to bring all this AI madness into Java? I mean, absolutely. It’s it’s amazing. It’s very cool that Python community had influenced them so much. like LangChain4j we should admit wasn’t created from scratch without any prior knowledge right there was python lang chain and it was an interesting project and it is so cool that you can take python code and be inspired by by it and create a new API in Java which is actually very robust and comfortable to use as well as springi obviously they have slightly different paradigms but well spring team is very well known for designing good APIs. Essentially, my whole life I didn’t like only one of their APIs. It’s Spring Security. Now,

[5:04] It became much much better. Is there more that could influence languages in both directions? Of course, we are at the Java event. We are all Java lovers, but yet there is no good or better language than another. Each has its purpose. Do you think we should be influenced with Java more by other languages? I think that diversity is very good for any sphere. The more diversity you have, the better product that because the more inputs you have and you can adapt your output to this input and this is very important. So yes, it is very important to look at other languages, take the best part of them, implement them in Java when feasible and it’s totally normal to discard ideas as well. Like I’m not a huge fan that we discarded the idea of string templates but they were considered insecure for whatever reason.

[5:58] So we discarded them and that is a good thing like we can make decision based on data based on evidence. So at least we have this idea if not other languages we probably wouldn’t have it and string templates they can reappear if they find a better solution on how to implement it. I really hope so. Simono de came to JCON as a speaker about the impact of AI on our learning capabilities. Should we use AI to learn? And you might guess the answer as it’s the answer to all software related questions. It depends. As with everything, right? It’s basically about the fact that if you are learning new stuff then there’s of course this risk of becoming too over reliant on it and that you are simply copy pasting what you find online and how are you going to judge that if you don’t have any base knowledge on the matter yet if you’re learning the stuff how are you going to evaluate if the AI is speaking the truth which could lead to you as they say it you can fall asleep sleep at the wheel and this is basically when AI takes over and instead of that you don’t use it as a tool anymore. So AI is in control instead of

[7:14] You and that in time can seriously harm like critical thinking problem solving skills etc. So there’s definitely a big risk. I just started an experiment thanks to CLI to help me. Perfect. First thing that happened was documentation that it was based on was probably outdated. So my dependencies were already wrong. So no, you should not trust it and really use it as a step by step to guide you through a new new project. Maybe my recommendation is to first of all find your source of truth. And with that being said, AI isn’t a source of truth because it’s not written by an Arthur. It is not reviewed or evaluated by peers. it is a prediction tool. So you need to find indeed your source of truth from somewhere else from someone you can really trust. more tips that you give during the talk about what not to use, what to use. Yeah, we really point out that you can of course use it to inspire your ideas because it’s a great idea generator and it can actually benefit your critical thinking if you use it to reflect on where you’re currently at.

[8:27] What is your current thinking process to give new perspectives but you need to be so aware that you are in control and that you’re doing that you not let AI take over that you first try stuff out yourself make mistakes that’s fine because you it will gain a deeper understanding of the material right but after you tried stuff out and you tried multiple things and you’re finished either because you’re completely stuck and you don’t know how to continue anywhere else or because you’re satisfied then include AI to give you either a possibility to untangle or to give you new perspectives on how you can simplify it, how you can do it in a different way. So yeah, always think first, do first and then invite AI to the table. Much like a tutor, right?

[9:14] Like use AI as a tutor who will never give you the answers straight ahead. they it always comes later and again and tutors will help you plan your learning, structurize your learning and give suggestions on how to continue. I do a lot of writing. What I found very useful was copying a block into chat and give try ask for a summary. Oh yeah. Yeah. And then what’s missing? other uses you see as a programmer or different use like writing or something else? yeah what I just mentioned a little bit like a big advantage of participating in this course is the fact that you get a tutor and this tutor is making a planning for you for the duration of the course. It’s structurizing your learning like what should you learn first, what should you learn second and AI can also do that for you right it can really tailor like where you’re at now how should you plan it different like even it can even make an agenda for you basically and with every session you do it can grade your current like progress and say okay then the next session should be or your next learning episode should about

[10:24] Be about this and that and summarizing is a great way I think but I do notice and I also read that back in articles and from personal experience that summarizing on the bigger scale is nice but again in retrospect. So you first kind of need to read the article yourself because the devil’s in the details and the hallucinations are as well. So you need to have a general concept of the article that you’re feeding the AI before you blindly trust the summary that comes out of it. Thanks. so we’re now talking about code generation with LLMs. How do you look at Java and AI in a program? How we can use AI like LangChain4j using it in a Java application? How do you think those are already good? Do they replace what we can do with Python? I think we’re very much at the starting phase of that, right? Like I think for especially if you go to conferences like this a lot of talks are about it but if you talk to the people in the hallways not a lot of people are actually working with it currently. So I think it’s a little bit hard to say and I feel that there are a lot of green projects around

[11:35] That haven’t really been proven in production yet. So it’s a little bit early to say if this will really have the big impact that we’re all talking about I feel. but there are definitely a lot of ideas going around. So yeah, but are not a lot of these bit overhyped? I think so. I think so. I think there’s a lot more to it than simply vi coding your way to make a nice gooey and do like this idea generation because it can indeed if you vibe code or if you just generate like this guey to pitch your new innovation idea, it can work with that. But is it really sufficient enough for production use? Is it really secure enough for production use? hallucinating all around like I don’t think we’re there yet. Steve P used the hallway track where you bump into all different people to learn about the AI interest and feelings of Java developers. But it’s so nice to catch up with people and one of the things that I’ve been doing and one of the things we’ve been talking here is trying to see whether we can start to build some AI Java workshops or even on conferences because we know that lots of Java

[12:57] Developers need to learn about AI. there are a lot of AI tools out there and so what I’ve been talking to with people is that can we bring that together not so much as a conference but more maybe as an unconference or something and get a bunch of the AI vendors all the different tools guys to come along and share and invite Java developers to come and meet these people and learn what’s going what’s going on because AI is a different beast and Java developers like everybody else got to learn learn how to use it and it’s moving very fast so for me the most exciting been it’s been but it’s been just how willing people are to entertain the idea of doing something where we bring the vendors and the experts and the developers together to share knowledge. How do you look at this AI evolution? Are you scared or you fascinated? What’s your feeling? well, so I’ve been writing a lot about AI from legislative points of view because there’s a lot of legislation happening. There’s lots of legislation in the cyber crime side as well which is I’m very interested in. So from an AI point of view generally I’m I’m not

[14:06] Skeptical in general but I am skeptical of all the people claiming that using LLMs to generate code is an easy thing and I really don’t believe that. There are certainly people coming up with AI tools that can generate code but they’re not LLMs. And so I think there’s a bit of a hype, but I use LLMs all the time. I mean, for almost everything, but in terms of generating code, they’re not my choice because I spend too much time debugging. When and if somebody can get an AI AI to reason properly, not statistically reasoning, then it’ll be a gamecher. That may happen soon, that may not. In the meantime, I look at AI as being an enabler for everybody. But it’s one of those things I’d say as a developer, people say AI is going to get rid of your job, but it’s like no, that’s never going to happen. What is very likely to happen is that the developer knows how to use AI tools. And that doesn’t mean just generating code. that’s like understanding code or teaching you about particular new features or using the AI as a helper that makes you more powerful and that will get you a

[15:25] Job and that could easily put somebody who’s not that skilled out. So I think that’s what I see. So there is a disruptive thing going on here. You know, it’s going to be interesting this time next year when we’re here next year, how much of ch how much has changed and how much has evaporated. And you focus also a lot on security. So what could go wrong with security wise if you generate codes a lot? it’s it’s unfathomable, right? So what could go wrong? the code itself is almost all the code that’s generated comes from the training data and the trading data is public data. So what you’re getting is effectively the average of the public code out there. and you got to remember that LLMs don’t look at code as code. They have no understanding of code. It’s just text.

[16:15] So actually it’s how good the doc is that goes along with the code for it to work out what the code is going to be. So poor doc or old doc can get dms confused and can generate code that won’t compile. That’s quite common. and then now we’re talking about looking at the world where we have agents that generate code. So apart from my skepticism about how often that will actually work, think about the power you’re giving an agent. If you say, “Oh, I want something posted on Twitter.” And you say to an agent, “Go figure out how to do it.” And it’s got the power to learn how to do that for sure. But then generate the code and run it. That’s very powerful. And if you give that give the bad guys get in their hands on that or you just, you know, make a mess of it, you’ve just given this thing autonomy over you, it’s you know power of you. It’s just really scary. So don’t do that. So we can agree you’re a little bit scary too. It is very scary. it. We are very lots of people are into the hype of this. I’ve done two talks last week which was about how AI can be weaponized and how AI is used against us

[17:30] And one of those was about how AI how the bad guys use AI to weaponize your code. So there’s an enormous amount of possibility for the bad guys and because we’re learning we’re behind them and they are very expert at this stuff. So we are entering a new world where for a period of time we’re going to be less security capable than we were. A bit scary message to conclude but something we need to keep in mind. Yeah. Yeah. Absolutely. Sandra Algrim, cloud developer advocate for Java at Microsoft sees a lot of benefits of using Java for AI. If I want to use AI services for my productive environment, obviously I don’t want to switch the language. So I want to stay with Java and using a framework like LangChain4j, springi or semantic kernel for Java will reduce the friction if for instance the model API changes or if I use need to switch the models because apparently I know we all love specifications but for AI models we don’t have something specified yet everyone does what they want to do. So it makes sense to have some sort of web. So, so that’s why we’re doing an introduction and also compare a bit LangChain4j, Spring AI and

[18:45] Semantic kernel for Java. Which is the best? Well, there’s a total biased opinionated question. So, if you ask my employee, you have to use sematic kernel for Java because it’s going to come up with support and everything. But for me, the communitydriven version of LangChain4j always drives me and I feel the community is very rich here. But apparently we also have Spring AI which is backed with the Spring team. So all three have their market I guess. I think I saw the first demos of this at Devox two years ago and some vox we already talked about this today. It’s amazing how quickly the Java community and Microsoft and others have built these libraries to make AI so easy to use in a lot of projects. It’s impressive. Right.

[19:33] We have Dimitro who’s like obviously doing the most job. Wonderful. Please keep on working. And then we have Julian Dubra my manager from Microsoft. We have Julian Defro from Google and you know all the other Red Hat and IBM colleagues and also from Greater Maven. We have support from everyone. So I really see this is driven by the community and I also like you know the buffs for instance that we had at Devil’s France just a month ago. So yeah, I feel the communication channels are very open. So that’s something I really value about language. Microsoft also has a lot of these services that they offer. Are there new things coming from Microsoft in this respect of services that you will offer or what are the most popular things at this moment?

[20:18] So we do have Microsoft build coming up in two weeks. I didn’t plan for this advertisement but yes there is build coming up where we have a lot of announcements to be make to be made and since we are putting this live I can’t say much because I’m not sure what what’s what you can announce yes because being an advocacy you know we always know up front and then I need to be careful what to say and what not but I just got pinged by another community member that my audio support pull request for language really needs to make it being merged so I will work on this night. So, hopefully we get that soon. Okay. Audio support. So, what will be able to do like using Azure OpenAI and OpenAI like just talking to so like speech to text and text to speech. Marie Grigleski talked about Genaii. So, she is the perfect person to explain it to us and how it differentiates from other uses of AI and LLMs. We also discussed the power of small models that you can run locally on your own machine and the community AI work. I’m here to do a talk about building a data pipeline. I call it chat

[21:30] GPT but essentially generative AI types of pipe data pipeline experimenting with it and also specifically too with rising wave stream processor and then essentially just see how data pass through and kind of construct you know taking the output from that stream and then construct a prompt and send it to LLM and see what happens kind of thing but essentially yeah it’s a genai type of talk so and the genai is based on data on text file where does it get its knowledge? Sure. So for now, well, for LLM, I’m just going to use an LLM u let’s say like open AI one of the you know large language models the GPT 4 40 or 01 or something and so and where does it essentially gets trained you know like you know what whatever data it is trained on by the GPT 440 but I’m also thinking of because it’s kind of going meant to be a bit more life coding I can also do it like locally using O Lama and just interacting let’s say I currently do also have like llama llama 3 that I connect with and but it’s everything is kind of the smaller model so I can run it locally so yeah so

[22:46] It’s just basically based on those publicly available models that I download so locally as well so yeah yeah yeah I think the these things are really great I was experimenting with a chatbased conversation about setting up a new project. I was looking for a way that I could translate these podcasts to another language or just have a transcript. It’s amazing what you can do with local models nowadays. Yeah. Yeah. Totally. Yeah. Local models too. They I think it is very true. Local models tend to be unless you have very powerful, you know, processor and also very you know big disc space to hold one of the larger models. Otherwise, if you do the smaller ones, they tend to be less accurate. But still, for the most part, I think it would works pretty good if you are just not going to ask something too specific or too obscure kind of thing. But, but it is actually I do have to say if we use chat GPT this technology correctly, it does help, right? A lot of things to it can do things quite nicely like sort through a lot of documents and extract information that you want kind of like

[23:54] That. Yeah. But as such as we also know too these LLMs this chat GPT technology is not that reliable in sense because they are non-deterministic the technology is more fuzzy logic so it’s just the nature of it too so but otherwise too let’s say you want to write an essay sort of a oneshot deal right it’s actually can do its job quite well like you said yeah the only thing I was struggling with that it doesn’t recognize the name Foojay it makes something very strange out of it but I probably train it to recognize it. That is very true. Yeah, it is true. If you do anything specific and it doesn’t know then yeah then it won’t know. It’s kind of people like to say like garbage in garbage out. So it’s more sort of like what you train it with that’s what it knows about. Yeah like that. But it is true there are good things about it and also kind of be cautious to about things that it may not work too well. So yeah. should I do AI development with Java because everyone thinks Python is the way to go but I think Java is just as good you know actually that’s the thing right us too we are so involved

[25:02] With the Java ecosystem to me I think if we want to be serious about using Java or kind of doing this type of AI operations on a larger scale on a more kind of complicated kind of scenario I Java will shine versus you know let’s say Python. I do have to say too is interestingly I was in Hong Kong like last week actually visiting visiting my sister and then I happened to speak at the Hong Kong open source meetup group and they also at the same time invited another speaker happens to be also from the US and he’s quite well known he wrote the book about polars which is a data frame library in Python quite popular now but actually he’s also an expert and he was he did tell me that basically well don’t quote But we were chatting and he did say well when it comes time right you want to do something quickly not more demon demo thing it does a great job Python but if you want to do something more serious like doing a production level things that can scale handles concurrency Python is doesn’t do a very good job we all know but this is also where Java can shine so well the

[26:14] Distributed systems the industrial strength I think to me right I think there are already effort in the Java community within even Java champion within ourselves. We feel we have so many experts we should really come out now and promote Java as the choice of language platform for doing serious type of AI operations. I think Java will shine too just because it handles distributed computing concerns so well. Yeah, as we all know. So and we already have very good Java libraries and I know lang for LangChain4j have experimented with that. Do you use any library in your project in your presentation? Sure. I actually I’ve used Spring AI so far. Although people say, “Oh, Spring AI isn’t quite, you know, the thing, which is true. It’s still kind of new, but I think it has made a lot of progress within the past year of or so since it came out to support Genai type of operations and it’s model it seems to be good and now it also integrates too with model the model context protocol which is like for interacting with LLM standardized way.

[27:18] MCP as such is more for lower level kind of thing and it’s perfect for Java if you want to write you know a library agent library for example it can do so well so spring AI is what I’ve worked with so far but I also did a talk with Daniel from Red Hat and he works with Quarkus Quarkus itself also has come out quite strong to have integration really well with LangChain4j so did spring spring AI with LangChain4j as well so I feel too in time these libraries will get caught up too. I mean don’t get me wrong I mean Python can do its job all that stuff but the language platform itself is not as ready for the real kind of industrial strength kind of thing but I feel Java should be able to get caught up. Yeah you mentioned Springi is quite new but isn’t this whole thing quite new? Yes actually you’re right dear Frank everything is new. Yeah, even we say okay it’s new but this Python isn’t that much you know older but it is very true Python did come out first to do handle a lot of the data thing right and I think it’s because Python does have an advantage of easier to do you not worry about types you know being

[28:28] More kind of not you know not the static typing more dynamic stuff but it has this drawback but it is true it’s more easily adapt or kind of you know take taken up by less people well people who are data data centric data scientists tends to be paying more attention to the subject matter which is the data less about programming so I think it’s it’s just that is the nature of it Python is more easily anybody can pick up but when it comes to serious stuff I think Java is the one it requires a deeper understanding I mean from the coding computing point of view so but you’re right too is even Python isn’t that new that old libraries yeah but isn’t that amazing that the whole Java community picked up something so drastically new like like this whole AI stuff and created lang chain 4G and open AI and Spring AI sorry within a year I think we went into from nothing to a very reliable productivity quality tool like like blank chain 4J I totally agree with you I think it’s also reflects on the capability of the engineers in this ecosystem I think it really shows that people understand

[29:40] Truly deeper the computing needs and stuff. So they are able to actually get caught up quite well and they can talk deeper into these concurrency matters distributed computing a bit more on that’s true the lower level technical details which is so essential in building any type of computing platform. So I think that’s why I feel I’m fully confident with the level of intelligence people that are from the Java community I think we can definitely be able to come up with something quite strong you know in no time and exactly like Frank you said we started get getting caught up and it’s been great yeah not that long but still I think yeah we’re able to put together these SDKs and that can do some amazing thing so yeah yeah yeah Jonathan Villa of Sonar gave a talk at JCON about infrastructure as code. I wanted to learn from him what I can do with code to run my application, but he also warns us about the dangers of AI when using an LLM chat to assist us. Well, basically in this talk, I’m not really covering Java, but the different artifacts that a Java developer in the end uses. So we need to deploy our

[30:53] Application in Kubernetes in docker or in any platform. So we usually combine our java files with kubernetes jaml for instance. so in some cases people think that code quality only refers to the Java code but also impacts infrastructure as code artifacts secrets hardcoded nonspecifying the limits in a Kubernetes u pod. so there are a lot of best practices and issues to check that with the right tooling and methodologies. We can also have high quality in infrastructure as code files. Okay. AI is a hot topic. Is AI involved in infrastructure? Obviously yes. Yes. We these days we can generate docker files or kubernetes files with AI prompting just create the deployment files for my application.

[31:58] Surprise. It’s not always as reliable as we would like. So definitely we need to check we need to understand what it is doing. So not committing blindly we need to understand we need to think that we spend 80% of our time reading code. So it’s a lot definitely we should invest time on checking that the code and the infrastructures code that we are adding up to our repository is in high quality and it doesn’t contain errors images outdated images with vulnerabilities or even not setting the right properties when we are putting a pot or a deployment or whatever. Yes. And secrets is probably also a hot topic keeping your secrets out of your repository or your flow. Yeah, because the problem with secrets well from the sonar telemetry in our different tools it says that hardcoded secrets is a top issue and I assume that the problem is well we all know we should not hardcode secrets right but why do we have that many projects with hardcoded secrets well I assume that it’s because we follow the natural trend so it’s like we create a proof of concept. Yeah, this is

[33:24] Only for testing my application. I’m putting the secret there because yes, it’s a test database. Uh-huh. And then this evolves and one day it hits production and yes, we don’t recall that we added some secrets in the code. So, we need tools or a lot of eyes in order to check those those secrets. So yes, it’s it’s it’s a big problem detected even in infrastructures code and also in programming languages like Java or TypeScript or Python. I think we all made the mistake of putting test code eventually in production. So your tools are helping to fix these errors or find these errors or be alerted. Yes, definitely. so in this case owner but it’s it’s I think it’s a reference tool but obviously there are other tools that you can check the depending on your needs but these tools will help you from the very beginning in the SDLC. So from the ID you will have a tool that it is checking your code. It’s analyzing your Java code, Python code or your Kubernetes, Docker Terraform or cloud formation code and it will show the issues in terms of security but also

[34:47] If you are not following best practices for instance if you are not setting the memory limit for a deployment for a bot well it’s not going to break anything but eventually you will have a bad experience with that. So it’s also enforcing best practices. and in some cases for some rules also sonar provides a fix using AI that will help in order to be faster in order to fix all the issues. But basically the difference with other tools or with the current trend is that sonar is a deterministic tool. So if you run the tool 1,00 times with the same code, you will have exactly the same answer. and this is super important. compared to other tools that are using AI in order to detect issues that are not deterministic and they are going to give false positives now and then. So it’s it’s important to use this kind of tools that can give you the confidence that you are always sure that those issues are issues. In the next podcast, Simon Martineelli will tell us more about HTMix. But during the interview, he also talked about the influence of chat interfaces on UI development and he

[36:07] Gives a nice description of MCP. just before we started this recording, you mentioned web development. Is that UI development? What did you mean? There was a talk from the CIO of Microsoft and he said don’t invest in UI development anymore because everything will be chat. So either language or typing it doesn’t matter but the user interface in the next few years will become chat based no longer form based where you have to fill out many fields just chat with the application and so that means that the UI will may become obsolete. I don’t believe that but that’s the prediction and also the business logic may be go away somehow because validation and understanding the data will be part of AI. but what will become very important is your database. So maybe as a tip already take care of your database make it consistent make it fast make it accessible and then you’re prepared also if that may happen or not. Yeah.

[37:11] Can I be a little bit scared that AI will be in between everything? We don’t know what’s happening in these black boxes. Why AI decides to return data in some kind of way? I’m a bit scared. What do you think? Yeah, me too. So very scared in fact. So because you really don’t know what happens and it’s not predictable. So the problem is how depending on what you do with it, you get a different result. You even get the different result if you do twice the same thing maybe. And that’s a bit scary, right? That’s true. Yeah. I find it very hard to believe that data of data can be handled by a system like that. Yeah, me neither. So we were talking with Lobby and Pasha as well. is also another speaker just before our talk here and he also said hey we learned over the years that the computer exactly does what we want so we if we do it right the computer does always the right thing he’s never wrong and with AI that completely shifts right because you don’t know if it’s right or wrong by the end so but I mean in my opinion we are like in the kindergarten of software engineering

[38:29] With AI right either it’s AI integrated in application even is it’s AI with chatbased or AI used to generate code it’s it’s really really early stage and what we see currently is probably not what we will have in a few years right so you’re a bit optimistic about what it can bring in the future I hope so because I was trying out like Tunai and Replet and augment and what else copile and all these tools And I was a bit yeah disappointed so to say because it didn’t do exactly what I wanted because probably also I was using frameworks that I know by heart and it wasn’t the way I would do that and to bring AI to a point where it does what I want. I need to provide a lot of information and if I have to write all the information in the chat for example then maybe I’m faster if I just programming or even if I have the old forms because going back to UIs are no longer relevant. If you have people that are working every day with an application, they are can do that blindly, right? They know it by heart how it works, what they have to do. And probably that’s much more efficient depending on what data entry job do you

[39:47] Have. you mentioned databases are important. So having a good structured way of how your data is organized. You’re also a big fan of juke. are those kind of tool important here? yes, I think because if you want to provide like I mean currently model context protocol becames new or it was introduced and I know already some companies that are planning to create model context protocols for their own data and somehow you have to implement that the API and if you already have a relational database you can use choke and then you have type safe access to the database and that’s some place where I see that could be very beneficial to user. So, can you explain in two sentences what is MCP? Yeah, I I’m probably not the right person to ask that question. That’s right. But now MCP is the idea to have a standardized protocol to interact from a chatbased or aic AI with something else, right? So you can create an API and the API is described with some description like open API or SOAP or whatever we had corpa maybe so in that and then you can for example if you use cloth desktop you

[41:11] Can integrate the MCP server there and say okay that’s MPC server and then cloud will inspect that API definition and see okay these are the capabilities of this API and now if you chat with that Claude will see oh that’s probably the topic that this MCP server can answer and then he will call the MCP server. So it it’s much easier way because before we had like rack as a possibility to inter or to add information to an agent or to a chat but this is very complicated because you have to implement that you have maybe have a vector database and stuff like that and with MCP you just can wrap your existing API or you can wrap your data or you can wrap your documentation for example. So Marcus Hellberg from the VIN team did an MCP server for the VIN documentation that you can integrate. So the chatgpt or the chatgptt for example will not just guess how vin works it will consult or try to figure out based on the documentation. Emily Young was another speaker at Jcon with an AI talk.

[42:17] We talked about LLM hallucinations and how to prevent them and make unit tests to validate LLM interaction in a correct way. I’m here to talk about validating LM’s basically I share true story how I lead a team to basically come up with some metrics and to validating basically a few LLMs including basically the granite IBM granite code instruct basically for enterprise Java LLM. is yeah because LLMs they tend to hallucinate as they call it and come up with answers which could be correct or not correct so how do we tackle this yes exactly you nailed it because LLM like is nature to do the wholeation so is sometimes you enter something and then repeating the whole sentence back to you so basically and the other interesting bit I find out actually it’s not necessarily generally LLM have more data and give you better answer. So basically in my experience right so okay this is a good LLM and then like the did the more adding more data and did the more training and find out actually it went backwards. So it’s basically it’s you do need to come up with a good stable kind of testing framework to make sure the

[43:44] LLM you are going to go with production is the best is the best performance LLM. But testing an LLM is difficult because yeah it never comes up with twice the same answer and if you write unit tests we really expect a specific results. So how can we handle this? You are spot on. So this will be in my talk basically is for testing different things you need to come up with different approach such as like for the unit test is like every time you ask to generate a unit test the first thing the test method may be different and all variables with different actually things you don’t really care about like they might be different however you won’t be able to use like a ground of truth you can’t really write your own unit test LLM or come up with this answer. So with that for unit test generation all you care about is a good code coverage right is that natural Java code like because when we use basically code assistant want to generate some code test basically similar to the human written test right I’m a developer if he is my assistant I expect to write something I can accept so this is basically for is you we use

[45:10] Basically GCO for the code coverage and also for the Java network this kind of metrics make sure like the generator Java code is readable and makes sense and the other another thing is the one like you don’t really want 100 test generator for one method right so also is a kind of some other metrics we came up with for this method if is for the if I’m the tester I will want to write probably five or six. if it’s like a LLM give me a 100 test result, that’s pretty no to me. Yeah. So even when using LLMs to generate codes, you still need to be sure what it’s doing, validate what it’s doing and checking if everything makes sense. Yeah. So also the for the code completion basically comment to code that kind of thing. I can’t really use like a code coverage metrics anymore for that. Basically also in my including in my talk we came up with kind of some ground truth kind of the testing basically we nail down the props and then we expect some similar code but not exactly same code. So we use a different matrix like a code blue to see the basically similarity kind of check and it’s not exactly like 100% match if

[46:35] It match like 80% so that’s good enough code. So this is kind of the how to test the generated code. So there’s another aspect basically you also use LLM to summarize the like the class. So that’s kind of sometimes the class maybe is a kind of very long is a again is the LLM will give you different sentences all the time right you can’t really use the true match exactly also you need to do the similar check for that also is a kind of is tedious for like human beings to write like a huge paragraph to explain things and etc and also LLM may be explained differently. So with that we use both human in the middle kind of intervention that kind of matrix plus like LLM as a judge. So this is another kind of good matrix if LLM can do use another LLM to judge the testing LLM kind of the result also is quite interesting matrix as well. Yeah. So you have several LLMs competing with each other to give you the best results.

[47:51] Yeah. Yeah. Exactly. Also another interesting thing I find out actually the judge LLM judge may not be the best LLM to do the work. similarly applicable to the like football judge like the also is some other kind of the judge maybe he cannot do that well himself but he knows like what should be covered so we find out that’s quite interesting as well yeah how do you look at how good Java is with AI depends on how good the LLM has been trained with Java kind of the knowledges and etc So the LLM I have been working last year with IBM research. So basically I personally like searched up a lot of basically the Java like the pages and etc. And then I purposely like get that data to train LLM. And then also during the testing we find out LLM is not a particular good at some area.

[49:02] Actually I led one really brand new initiative in IBM. I teamed up with some other like a Java like experienced the Java developers. We wrote some data use that data to train the LLM and then we can see the difference use my basic frame framework we can tell like after the LM trained with the special data we wrote is getting better. Yeah. again is basically depend on the LLM whether the LLM has been got a special training in some languages but u the what X code assistant for enterprise Java from IBM I have been working with is did really well on the Java kind of the unit test generation code completion and also the code explanation as well that’s it for this episode in the Next one you will get the remaining JCON interviews with a lot of tips and tricks to be a better developer. Thanks to my guests and thank you for listening. Please subscribe to the Foojay podcast in your app or on YouTube. Keep an eye on Foojay.io for future articles and podcasts about development and everything related to the Java world. See you.

[50:26] Give me the friends of OpenJDK.