Foojay Podcast #17: Execute Java Code with TornadoVM

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] Hello and welcome to this new edition of the Foojay podcast we’ll talk about tornado VM which allows you to run Java code on CPUs gpus and fpgas all your news about OpenJDK today we’re talking about the tornado VM project it’s a programming and execution framework for offloading and running JVM applications on multi-core CPUs gpus and fpgas with the same code some of your existing programs can be executed hundreds of times faster now we’re going to go and introduce the guests yeah hello my name is Juan fumero I’m a reserved fellow at the University of Manchester working on compilers private premium models on runtime system for heterogeneous Hardware so I joined the University of Manchester back in 2017 late 2017 right after I finished my PhD and I joined directly in the tornado VM team since then I have been working on VM as a well now I’m in the one of the main committers of the project and I also doing some work in the software architecture in combination with funds and Christos I have experience our modern 10 years experience working around GPU compute for diverse compilers CC plus white compilers with open ECC

[1:24] And then I jumped to Java back in 2013. I did ipsd in the Java side and I did something similar with tornado in thus it kind of the father for me in some way the precursor of tornado VM in my PSD I focus not only on job but also other JVM based languages for R and Ruby to run on gpus as well and since then well I have been staying with the javasai hello my name is and I’m an Astrid professor at the University of Manchester and also chief engineer at KTM I have been doing research on Java and television machine for more than 15 years mainly on compilers and memory optimizations at the various levels of the JVM and I’m leading the autonomy project in the beginning which is almost 890 years now throughout the years who have been developing and improving internal VM and trying to help developers get the maximum performance they can get out of the absor machine I mean my past life I’ve been working in that Oracle and Intel on the hardware and compilers and they will continue that work now at the University of Manchester hello I’m fanastad koplus I’m a research associate at the University of Manchester and I

[2:45] Have joined the tornado VM team four years ago my ex expertise is mostly on the field programmable gate arrays fpgas so my in my PhD I used fpgas for accelerating storage my experience in Java and compilers is also coming from the time that I started the internet VM work and I have done some work also with install instances in the cloud for Hardware acceleration of different Java applications yeah my name is Jacob Demko I’m a Java developer since 1999 I’m a Java champion and I’ve been interested in performance optimization since I was doing Assembly Language programming on the Amiga 500 so it’s been a few years A couple of years ago tornado VM popped up on my radar and I found it very interesting and then I think some sometime this year or was it actually it was late last year it popped up again and some of the things that I was missing the first time I saw it was now implemented so I thought it was about time to have a look at tornado VM again so that’s why I’m here mostly as a guest and as a an interested user so yeah I have just been able to run one of your tests today with

[4:05] The tornado VM and I got a I think 1100x speed up on one of the cases here so that was pretty pretty impressive and I’m Eric costlow I’m on the Azul product management team and I focus on the intersection of java and security of how do we make our applications run fast and how do we defend them against modern threats all right now we’ve had a couple Foojay articles about tornado VM one in particular talks about the role of Hardware acceleration for tornado VM and yakub how you got what you introduced as the about was it eleven hundred percent that’s pretty significant who wants to go through and just summarize what we talk about in the Foojay article about how tornado VM works and when and where you use it sure absolutely so let me summarize the tornado VM in an answer it is an open source software plugin to different JVM distributions that unlocks Hardware acceleration on multi-core CPUs as you said CPUs and fpgas the main objective of tornado VM it is that the hardware acceleration was not by by Design included in the JVM therefore we wanted to audit it in the programmer friendly way and we have many different

[5:24] Features exposed internet VM so we support portability across different Hardware vendors portability across different Hardware device types like multicore CPUs gpus and fptas and also through the interoperability with the graph VM we can accelerate we can invoke from different programming languages that are implemented on top of travel the Java accelerated code so how exactly does that work we have a normal Java application that we run probably something that’s got to be a little more advanced than hello world but let’s say I just took something like Apache Cassandra and I drop it into tornado VM does it automatically offload onto gpus or do programs need to be Rewritten to indicate which portion goes onto the GPU in general the way that Toronto VM will be integrated either in some user code or in another platform like a Cassandra or a passive link that we have done already the integration it differs essentially acceleration of only videos or fpgas is not always successful meaning that this kind of devices they are very good at particular types of workloads so there is some help needed by the developers in order to

[6:46] Identify which part of their application can actually benefit benefit from Hardware acceleration therefore we have this type of different devices because it’s one of them it’s really good at a specific task now if somebody starts from scratch an application or they have their own application or their own libraries the idea it is to use the apis provided by tornado VM in order to offload their preferred tasks on the accelerators if we want to integrate a larger framework like a Cassandra or fling or parties Park ultimate VM then the storage becomes a little bit more complicated because this integration normally happens at a systemic level the idea for Toronto VM is that developers shouldn’t care too much about what’s happening underneath the same way that the JVM or the Java programming language was created with we write once and we run everywhere in the same Spirit we also try to carry it through this kind of let’s say new accelerators so a developer of Cassandra for example which is used to using the apis of Cassandra ideally should get this acceleration out of the box without even noticing that tornado VM it’s

[8:14] Supporting and underneath so for this kind of large-scale systems normally what we do we go inside and we do the integration under the hood so developers will keep using their apis as they used to now for applications that or Frameworks that we have never seen before this is user code then a developer can actually use our apis to create their applications from scratch all right now I don’t know very much about gpus most of what I know how to do is to play a game which means that I have the right driver installed and I just open something up like Minecraft where you can do redstone programming because it’s written in Java and is a great game so when you talk about Java developers being able to drop their program onto a GPU and tornado VM kind of figures it out first what’s the limit that your average developer should know about knowing what loads on certain gpus so that we can just get a feel for it and then what’s some of the more detailed information that they probably shouldn’t have to know yeah so that’s a that’s a great question so we try to hide in the tornado side as much as

[9:24] Possible as much as complexity as possible from the GPU programming side so you mentioned that for video games for example you just need a driver for programming more than video games you also need a natural implementation like Coda or opencl and tornado actually makes use of those implementations in fact there are actually already three back-ends so from the plumbing side we tackle the well we have been thinking about permeability for quite a long time already and we think to just use tornado as simple as possible in the sense that we used to program okay sequentially sequential programs and we want to do is to add minimal information to the sequencer program in order to be able for the tornado compiler or any other compiler to paralyze application to transform that sequential code into parallel code for example from java sequential code to opencl or Cuda code and then tornado will handle this transformation those optimizations data manipulation buffer creation kernel execution etc for you in the case of Minecraft for example if you have this code based in Minecraft and let’s say you want to do

[10:40] Some some Ray tracing for example and I I’m aware another Minecraft support Android amounts to do to fabric tracers in opengl so in a scenario for example they would integrated or VM with Minecraft then they didn’t have to write this opengl functions the Minecraft code base should still stay in Java and use another VM to program those functionality they will have projects within our team where we actually have implemented a ray Tracer in Java a single threaded Ray Tracer in Java accelerated VM where you can get real-time over 60 FPS of performance so if Minecraft wants to do the investment and keep the whole stack in traffic ecosystem and all the benefits that this can bring then anybody that would write let’s say some CPU kernels in opentl or opencl or codeine plug them to Minecraft now they can write them in Java which is easier internet will be able to the heavy lifting of accelerating those on the CPU all right so one of the things that I do know about gpus is they do a lot of really multi-threaded applications so whereas a CPU might have you know eight cores 32 cores or some

[11:59] Number the gpus seem to have like thousands of them so what are the real differences between a thread as I would think about it from a CPU programming and a thread as I would perceive however a GPU parallelizes it from the gbo perspective when we call thread it is like a lightweight thread on the GPU we prefer to call it instruction streams because actually they are culturally executed in a stream fashion like in lockstep it’s a bit unusual if you were coming from the CPU side how these are programmed I can’t give you an example so let’s say we want to operate with two arrays we want to sum up two arrays I think a plus b so the way to do it on the CPU let’s say we use C or we just Java multi-thread it we have split let’s say we have I know eight cores or 32 cores we give up each core a portion of the work to do and that could work right that can work on the GPU side we prefer to do in another way so we have specified the work to be done Patriot and then we just when we launch those kernels we just call multiple instances of that kernel and we map the input space in the iteration space of the input size

[13:18] With the number of threads to be deployed for example if we are running an application that has input size of 1 million elements we will launch one million threads obviously we might not have the resources to run all of them in parallel but the GPU driver and the GPU Hardware will take care of splitting this one million in smaller blocks in order to run as efficient as possible okay so yakub what are some of the workloads that you put onto to tornado VM like as somebody who uses it rather than than works on it you have your choice of when do I reach for a regular JVM and when do I reach for tornado VM and how do you really go about making that decision to be honest at the moment I haven’t gotten any further than trying out some of the samples that come with the tornado VM but the kinds of workloads that I would be looking for are definitely data oriented types of workloads where you have to scan through large arrays of data typically stored in memory are stored sequentially on disk so you load them in really quickly you pass them on to the GPU and the GPU can do its calculations really quickly

[14:28] So those are the kinds of workloads that I’m I’m looking looking to run with the with tornado VM I’m not sure if you mentioned but like one of the things that I found interesting as well is that it’s not just the GPU that it can run on but like you also just parallelize on whatever idle or not idle necessarily like CPU cores right so one of the other things that I’m looking that I’m hoping one day will be possible with tornado VM is a lot of desktop computers today come with at least two graphics cards right one built in and then an extra one and I’m hoping one day it’s possible to use tornado vm4 like desktop type applications but at the moment I think the installation process is amazing yeah maybe it’s a little bit the poly or like how to say streamlining or whatever for that to be possible but this is one of the things that I think would be really cool to couple it with the JavaFX and then we could see some really high powered Java desktop applications in terms of those desktop applications a lot of people have mac you’ve got either the M1 or M2 and Thanos you wrote a recent article

[15:32] On Foojay just talking about some of the test benchmarks of tornado VM and I think you saw something on your M1 MacBook what like a 738 times performance increase or something just really really high so can you talk a little bit about that and what you see this was a an implementation of the matrix multiplication in fact it was an implementation using the tornado VM apis by that time I was using also the x86 binary of the JVM built on my M1 arm that was going through the binary translation to run on the system I was impressed because it was the first time that I tried tornado VM on the M1 and it was quite straightforward in the things that we had to adjust into media VM it was just some things in the drivers because we have seen also that apple is deprecating the support of opencl I was a bit skeptical when I was trying that but finally it was supported and I saw that all our unit tests that they were running on my previous MacBook they were running also on the M1 it was very interesting to see that with tornado VM we can unlock this high performance as you said with the speed

[16:53] Up that I got compared to a single threaded Java application that was running on the MacBook but it’s also more interesting to find other things like for example the Energy Efficiency of these implementations because we are talking about the arm but this is something that is ongoing on our team to enhance tornado VM with this Energy Efficiency functionality all right now a lot of the data intensive workloads that we have like we talked about a tornado VM being really good for computation based and when we have to go with huge amounts of memory and huge amounts of data the majority of us are going to the cloud with like AWS or Azure or gcp where you just buy like an ec2 instance so what are you doing for tornado VM instances up in the cloud like are you buying the same workload types or how are you handling that the workload types and they’re trying to find applications or the use cases called the real VM it’s actually one of them the hardest part that we are working at the moment because it has been used throughout the years for many different domains from several development to desktop and development

[18:07] Is always I would say tricky but it’s always very unique in the way that the developers try to approach different kind of problems by using the same programming language so the expectations for at least the workloads that they want to run and everything that comes with it for example data sizes computational intensity and where Tornado can fit inside and help developers it’s actually one of the challenges we have as technology developers in order to find this sweet spot which is a sweet spot for a specific let’s say a group of developers that they develop other Java effects or they run through another framework on the cloud for some heavy duty computation in order to accommodate all let’s say this is our useful thinking for us it doesn’t matter if it’s like a MacBook Pro or a cloud instance our objective is one if you have a CPU or an fpga that can they can unlock your 10x 100x we’re going to the developer now the question is how it’s easier for a developer to actually use our system in order to achieve those SP davs this is an ongoing process and we engage also with developers to try to

[19:23] Learn more what are the workloads and how they use them in their framework in order to do some changes in the API or even add some features that are really important for the specific workload so to turn it in now it can be used on any laptop download it or even we have tracker images and also have tested on Amazon Cloud that have a gpus or an fpgas so we know that tornado game can run on these platforms and then we’ll developers how developers will utilize turn of them inside their ecosystem in this complex application this is a discovery phase for us and it differs a little bit from traditional C or C plus developers in a way that when could our opencl came they were very similar in how to use them like I said physical Plus Code or even was very natural for developers of C to call the code kernel in the drama world it’s a bit different because Java control machine has the Cabbage collector the memory management and developers normally don’t worry about those kind of things so if you go to harness a CPU we don’t want developers to reason about where this buffer will go how the garbage collector

[20:40] Will treat it is it gonna set fault I’m gonna have double pointers or how to have consistency how my four Loops should be constructed and this is something that we have been iterating a lot for example in our latest version of Toronto VM we have refactored significantly at the API because we found out that the original API was a little bit confusing for developers in order to distinguish the Notions of the computation and data management data movement between the different accelerators so this is a working proper this is the worker process also to find out this one’s just got it so we’re talking about the API that developers interact with to run better on tornado VM or maybe I recognize that I have that data intensive processing what is the API and what should I know about how to structure my application like is it annotation based is it inheritance based like if I want to write something to run really well on tornado what code paths or API do I interact with then what should I know as crystals mentioned we refactor the apis the latest release and we have been thinking quite a lot about

[21:50] This for a long time so one of the questions we ask in order to have a power application not just alternator but in general is first of all how do we represent parallelism so in the case of java an Acres of tornado we have two options one of them is through annotations so we instrument or we instruct code somehow with annotations to give the compiler some hints about how to paralyze things how to paralyze Loops that is the most expensive part so one way through internet is through annotations and the only way is through what we call a kernel API the difference is that the annotations as we I mentioned as we mentioned at the beginning is that we annotate code the Java code that is sequential one of the sequential code this is similar to for example openmp or open HCC models for cc plus unfortune and this is nice because we don’t have to reason about how to paralyze things that much we can express our computation in a sequential Manner and the turn and the compiler will do the magic somehow we do the best to map it but this could sometimes could be quite limited so we open recently another way to express parallelization

[23:07] What we call the kennel API and this can LPI is a very similar actually we borrow 10 from lower level models like opponents opencl and Cuda and you express kernels in a very similar way so instead of annotation of Loops we express the work to be done per single thread and then we would what we do is to Launch hundreds or millions of threads on the GPU this is the problem of how to identify parallelization then we have a second problem that okay we annotate Loops for example at the method level or we express kernels with the kernel API but we do this only at the method level so we compile the method level and that’s what will constitute over candle and the second part is okay along all these methods that I have from java okay from different classes could be which ones I want to offload okay so the first problem was about how to identify realization how to provide parallel Catalyst and this problem now is how I select different kernels to be offloaded and we do this through what we call a task graph so basically it’s a data structure that Pro tornado provides is very easy to use and you can build graph

[24:24] And you attach data and go yeah so as the engineer who’s writing my application I should know the task graph because that’s how I structure out my work to be done exactly exactly so basically a task graph is just how data flow is Flowing across different methods that you want to call on offload can I just return for a short moment to using tornado VM in the cloud because my day job is as an AWS solution architect I would like to try running tornado VM in an AWS Lambda in a serverless execution environment and I know that the AWS Lambda can run Docker containers and as far as I remember there’s a Docker container with tomato VM packaged inside of it and I also know or remember that with AWS lambdas when you make them a little bit bigger you also get some virtual CPUs and if you want to use those virtual CPUs for parallelizing work you will have to do it like manually in Threads but it would be nicer just to be able to let tornado VM utilize those extra virtual CPUs it’s getting especially interesting since I think the last AWS re invent where Amazon launched this snapstat so now the lamp does Java

[25:42] Lamp does will have a very low cold start so Java is really becoming a first-class citizen in the London world now since I’m a part of azul I gotta talk about crack which is the core that snapstart is based on I gotta share the enjoyment for the OpenJDK crack project do you really like when you run on lambdas do you get to use much GPU like I have no idea what CPU instance or anything that lambdas even run on there are no gpus or fpgas available through Lambda but just you get the extra virtual CPUs and I was wondering if it was possible to have a tornado VM utilize them someday I have no experience on Amazon and be cautious with my comment now but if those instances so poor opencl or any other compatible model we should be able to run right for example yesterday we did a quick test for one of our projects we plug in an open source implementation for opencl running with P threads on the CPU in a local image so that was fun we could do this so I have no idea about Amazon instances but yeah something will come from civil usually beat them under discussed in the

[26:58] Context of tornado VM normal people internal VM and they automatically think CPUs or fpga is but another fact is that since we compile Java byte code to an underlying representation for parallelism like opencl or PTX for Coda or Spirit essentially any CPU that supports those drivers like open cell drivers for CPU VM they can also get the parallelization right so if you have a loop and you annotate it and you run through tornado the tornado will parallelize for the 8 CPU course that you might have and on top of that those drivers they also apply Automatic Auto vitalization so if even if you don’t have a GPU and you have a parallel for Loop or I don’t know if it’s on a cloud or a desktop and you have a parallelism and you run through tornado the performance that we get normally it is even higher than Java threading because we stack up both the parallelization plus the auto factorization that some of these drivers can give you your starting point again it is a Serial for loop with an annotation so going back to Jacob’s question if you have a Lambda again played with lambdas but if you have a

[28:23] Lambda on a Amazon instance with some visual CPUs and there is for example an official driver installed on that instance tornado should work out of the box for example cool so I got the AWS ECS guide up here that talks about a bunch of instance types where you’ve got you know your P3 your G3 looking at each of these which of these AWS instance types do you look at and just think wow I want to run on that it would be really hard to Max this whole thing out like I see a P2 16x large has 16 gpus with 192 gigs of memory and a p4d 24x large has eight gpus and 320 gigs of GPU memory like which of these just feels the coolest for you for sure what we’re saying to get big numbers that’s a fact so as soon as we start you know terraflu territories will always very happy but in general it’s a trade-off so you may have a very huge GPU but you may be never be able to utilize it because all depends on your workload and your data set and we have done research that proves that in some cases it’s obvious you go for the high CPU highest memory because this is what you need but sometimes the winner

[29:53] And I’m quoting the world winner because this depends on the slas of the application they’re a little bit in a gray area so you have the same kernel you adjust a little bit up and down your data volume and then the CPU can outperform the CPU because you always have data transfers between the devices that you have to account so but there are a lot of fun if you have acceleration the bigger the better for us we somehow overpass integrated gpus same with the CPU also many CPU system for example my laptop has this Intel CPU and has an Intel integrated GPU it’s not that big that fancy compared to the Nvidia one but it’s usable and tornado I just can’t use it in a transparent manner the thing is there are some type of applications because we share memory that because we don’t have to you know perform this data transfers back and forth It’s Worth executing on that GPU can give you a high performance than the discrete GPU talking about those gpus you mentioned a couple of the languages and I know a lot of software Engineers you can kind of flip between different languages like yes people can write Java sometimes

[31:04] Maybe they’ll flip over to a node then maybe they’ll jump over to python but in the GPU space you’ve got like opencl you’ve got Cuda and just what’s the difficulty associated with writing a direct opencl or Cuda program like since you guys know how to write them how hard is that like something that normal people do or is that something where it’s like oh I shouldn’t invest time in kind of learning that unless I really am interested I myself I am very fan of cool dance that’s the first language but I’ll give you a language I learned then I swap to open select differentiate GPU language from regular language GPU supports a subset of what we call a lens right it’s not doesn’t support a whole C program or C plus plus program programming for gpus is an optimization problem and as any other optimization problem you need to know architecture details and gpus are different from CPUs so I can give you a very easy example reductions so let’s imagine that we have an array a plane array and we have we want to perform a reduction sum all the values so from an array my input the output is

[32:18] A scalar value now I want to run this reduction on C right or just Java I just write the for Loop and just some of the elements but now so the GPU or any other highly power programming language now reductions depend on values completed from the previous iteration so to perform a full reduction I should restructure my algorithms quite a bit and to put things even worse so data is not shared with all elements in the GPU so we need to be careful so we need to split data smaller chunks and is only shared in smaller chunks and we need to know which processors contains that data to put things even worse cash is not coherent so if we want access to direct to the right value we need to guarantee put a barrier to synchronize all elements this is just for a simple reduction right it’s fun to do it you can do many you know type of reductions and so on but tornado or Franco’s like tornado can do everything automatic for you got it so I feel like people can kind of pick Java up and you can go from like zero to like yeah I’m familiar with that I did it in a weekend a week or some kind of short time

[33:32] It feels like I’m not going to become super proficient at like Q2 programming on a Saturday it depends of the background so it’s very used to GP your programming for a while those terms sounds familiar ready for you right so having a good account and then going to opencl or one API the latest from Intel is similar right if you’re starting fresh you need to catch up with a lot of new terminology not just a problem inside but also the architecture side and that could be one of the obstacles to face GPU programming in general and if you target fpgas then it becomes more complex also because fpgs are way different yeah I don’t I don’t even want to approach that as a regular person even just if we return to these potential use case for desktop applications as far as I can see in the next generation of Intel CPUs we get a lot more efficiency cores for free even in the even in the small CPU lines or so it would be really nice to be able to just use utilize them as well semi-trans apparently with tornado VM and like my impression of the API so far is that it seems reasonably

[34:48] Easy to approach like you say Eric it’s probably something you can get the hang of in a weekend or so the API and the annotations for the Java side I feel like I could write a tornado VM application in a weekend but I feel like if I had to learn all the intricacies of a GPU to program directly on Cuda or opencl like boy I better clear my calendar for that well to be fair just to clarify so those models could an opencl usually offer libraries so many people approach first by using libraries Cuda has these many levels of deep learning machine learning Etc one of them for a ray compute is called thrust Cuda trust and that’s the way to go but similar to Intel one API or opencl so the way it should be through libraries then as soon as you require more expertise or do more fancy things you can tune low level kernels exactly yeah I kind of like the approach that tornado VM is taking though because I can write the Java code and I can have Java code work well you know we got the right ones run everywhere thing so you got right ones now run on gpus I feel like if you go straight to Cuda you’ve got like

[35:58] Right ones and then you switch your CPU and you have right again and also the deployment is a problem because you have to bundle these libraries inside your Java code and zip them and as you said as soon as you change the hardware okay then you are locked in Nvidia let’s say you go on openc opencl to have more it’s a variety of devices looking back using what you’re talking about with the difference between like an AMD and an Intel when I looked at that AWS sheet there where they say here’s what you can buy they told me number of gpus they don’t say is it an Intel GPU is it an AMD GPU so I could probably figure that out but I feel like if I code and I choose the wrong one or I choose the right one and then it gets changed that seems problematic if I’m having to go direct definitely and I think if we see the industry now both in the with xylinx acquisition they’re trying to create their own unified programming models and Intel also with their own lineups they try to create their own programming models and what all of them have in common is that they’re trying to create the singularity the single API which is

[37:09] Very good for developers and then the vendors will optimize under me so there are many initiatives from Cronus group or one API that they’re trying to bring even in the nonzalea old all these companies together to help develop personal harness this kind of accelerators easily so it shouldn’t matter if it’s gonna be an mdp video or an interview GPU or Nvidia CPU the same way for example now at least for us for the Java people the JVM guarantees that if you run on an Intel CPU you’re gonna get very good performance if you run on mdcpu and the reason for that because in the nmd or are the two different physics at the compiler level and the optimize the platform for all this kind of architecture and I think this is a good thing to have as a community this safety that the jbm is optimized for all platforms available it does to complement what crystals mentioned is really a good point even if we go to the root of a pure opencl opencl is a standard as maintained by Chronos group is a Consortium of many companies and institutions and they agree on the apis and each vendor can implement the API so what you can have

[38:21] Is your opencl program running on Intel and tomorrow running on AMD or any other vendor from different companies and the code will run so they guarantee function anality portability let’s say but performance portability is another thing right that means that maybe you have performance for one platform but achieving performance for another platform is very difficult tools with tornado we have a lot of work we have spent a lot of time a lot of our tuning digit compiler to try to you know work on the performance portability side especially on the fijias yes and I have tried as a person that I have tried the tornado VM on the AWS instances I have installed tornado VM the software both on instances that they have AMD and Nvidia tpus and I have seen that you need also to install the drivers although there is a in guidelines from AWS regarding finding the correct drivers to install but regarding the installation of tornado VM it is similar and smooth especially on Linux systems with a automatic and installer script that we have online also the same stance also for the fpga instances that we have

[39:38] Tried which is the F1 instances but coming back also to the previous question we haven’t tried on the instances that they have multiple gpus and the reason is that we couldn’t afford one they couldn’t give us the access to this kind of instances yeah that’s outside of being the university exactly got it the university doesn’t just spin up Cloud resources and then leave them alone like a lot of companies do and forget about them we have some credits which is good for research but the credits were not enough for the research part to give us you know the access to the huge instances that they have the multiple gpus which is something interesting that we are also very interested in we have done some work to concurrently access the devices at the same time given the number of gpus that some of those systems have you guys could probably team up with a group that needs to heat a building and in the future that we have in mind yeah this is the goal programs gpus are key player in into this kind of workload and Java to get a foothold in this kind of this ecosystem because now for example pythons and this kind of

[40:55] Frameworks are more widely used and GPU becomes part of the equation for the developer machine this is our objective how to help developers in Java do their stuff and in that moment with so many billions of other developers we’re gonna be able to hit a lot of buildings yeah I’ve actually seen the chat GPT guys talk about you know however many gpus they have and those guys talk about gpus a lot and you know now that Java code can run on gpus you know what’s going on in that space like why are the chat GPT guys always talking about gpus and you know Java has tornado VM like what does that bring to us that’s a very good question and I think now we’re going to a philosophical discussions about the these future like truth is that I don’t know how dpd does it internally but the majority of the machine learning Frameworks in AEI Frameworks are being used they’re using under the hold optimized libraries for training the models from Nvidia or from Intel for example they use a 1D name or put the DNA and this kind of libraries and normally in front of these libraries which are let’s say binaries or List

[42:04] Limitations they have some apis like Python’s tool actually allow developers to experiment very very fast and in example there are Frameworks for deep learning that they follow the same path for example there’s a Java API I can call a kernel being implemented through a Cuda library to run on GPU and this way of programming machine models or AI through a Java doesn’t differ at all I would say in terms of capabilities from the GPU perspective of course the more apis we are the more capable are still become our vision is that since the gpus are not or at least action gpus is not yet formalized within the machine and steps are being taken now through project Panama and this kind of new projects on how to formalize foreign functions which can actually help immensely this kind of formalization of calling these kernels we believe that another game can play a very key role into using our experience and let’s say our software in order to create such systems that it could be used in machine learning in AI world within the developer system so the same way we have a collection API Java

[43:22] To have a learning API that under the whole use terminal VM to get out of the box this DB acceleration yeah so Panama is one of the OpenJDK projects can you just give a rundown of what Panama is and why it matters to the work that you’re doing in tornado VM and to be honest we’re not expressing Panama to our understanding it has a objective of creating these foreign functions so I would say like a replacement for jni and also to formalize how to do a memory allocation of hip so going back to the problem or at least one of the challenges is that a telephone machine has compared to other programming paradigms when it comes to inject acceleration in Java since we have a garbage collector the users are used to creating a new object or a new array and this area will go into the Heap and the covers collector will take care of it it’s a plus plus no you have to manually allocate the data so the developers know that they are responsible of doing this manually so when you run on the GPU in that concept you’re also responsible of creating this memory and copying the data from the CPU site to the GPU site

[44:31] Let’s assume that we do the same in Java which we do with printed VM what happens if the Cabbage collector hits when the data are on a GPU and the garbage collector will remove the data in the whole side of the CPU site the GPU will never find out because there’s no way to find out what’s happening on the CPU and when the execution premises it will go back to update the data in the pointer will be gone and then you’re gonna accept fault interesting so if I can kind of paraphrase that just to make sure I understand it there’s without like a Panama or without the ability to reference memory there could be a situation where the CPU garbage collector walks around and says I can clean that up but later on the GPU comes by and says where is that exactly so what Panama does it is it formalizes how to allocate memory outside the Heap before Panama was unsafe operations so you’ll come with that safe library to do it manually and which has some risks now with Panama it seems you can allocate data of heat let’s say in a formal way through memory segments now the idea comes okay if I know as a travel

[45:41] Developer that I can actually do simple acceleration then I could directly allocate this data off Heap to solve the campus connection problem right since I know that this is going to be big arrays for acceleration in compute there is no point of having an easily here so I can use Panama allocate them both hip and then I’m responsible for that and as soon as you have Panama at the Baseline for that then we can start building libraries on top of Panama of a whole collection of a data structures that can be of him and this can really improve performance for specific workloads that they require a lot of CPU acceleration hence you want to get the gas collector out of the way of making your let your life more difficult yeah also with Panama we can combine the type of buffer with some of the optimized GPU buffer transfer mode for example we can do pin memory between the GPU and the CPU or we can do automatic you know a unified memory use the unified memory and the driver will take care of migration if needed back and forth simply they all use directly share memory for integrated gpus but that’s our kind of for talking

[46:55] And to having Panama so Panama will give us the possibility to exploit all these combinations in a more standardized way yeah and if this is one of the main challenges that in our opinion this kind of acceleration or at least using these hardwood accelerators has been a problem or at least a very big Challenge on the tpus because these problems they require the full community to come together as thick of a solution like how to design a garbage collector that can tackle data or Trace data on the GPU side even technically this is very hard to do now because you need the universal memory which is something fairly new and not let’s say proven yet and we are doing part of us we’re doing research in this direction and provide feedback to the community how that future would look like let me also compliment that when we allocate memory on Heap and we jump into the CPU for execution then this is frozen so it will not perform an actual collection otherwise we would seek fold every time that we run currently with only the VM yes to be more technical we do a critical section on the jni side just to guarantee that part all right so

[48:11] That’s some cool stuff I know you guys just had a release in February where you rolled out things like the task graph and some annotations so what are some things that people should do if they want to try out tornado VM the first thing to do is maybe to try the example Suite that tornado has that’s actually the past that some of the potential tornado users have taken already so we have a suite of different applications including applications for fintech or typical thing for machine learning and different domains like physics DFT and stuff and people try Okay this match is kind of the computation I would like to see I would like to do and then from there they tried examples that they studied the code and so on I would say that’s the first thing to do and then depends on the background of every person right so I mean we have people coming from different backgrounds and that’s the thing we try to you know combine them all we have had people or developers coming from GPU pure GPU domain so that’s why we have these kernel apis and we have people that are coming from the pure Java side without any GPU

[49:24] Background got it so if I were just the developer taking a peek at tornado VM instead of me going and saying how should I rewrite my program to run really well on tornado VM I take a peek get the samples I run the samples on a CPU I run a samples on a GPU I look at the difference and then I start taking a look and saying well what’s different about these and now that I know what’s different once I know it that’s when I start saying how do I write for tornado VM yes and then in fact since the latest release we have a new documentation site where we document all the programming model of tornado VM and we give some introduction how of the main features of tornado VM how to install it how to run the examples as you said and how to try the test cases the Tomato VM profiler and other core things yeah and one last addition of that the most important is to tell us when it breaks so to try to improve it and secondly as we know that Trigon VM doesn’t support the full Java feature Set because it’s not because of no VMS because the GPU for example has some limitations like performance system calls and stuff like

[50:36] That so there’s a list of things supported by Toronto VM so of course my regular application you have to make sure that we don’t use this kind of features and are there any major apis that people should be aware of to say oh I shouldn’t do this on a GPU like it sounds like process Builder or runtime.exec we haven’t already done specifically for apis but General reflection so anything that requires reflection because you need access to the class pointers which they’re not they don’t exist in a GPU or system calls that they need always support these are in my opinion the two main features that cannot be executed on the GPU and there are some other smaller ones but I think reflection and the system calls are the most important okay then most of the data processing they don’t use that so I think that’s that’s good good also worth mentioning we have some features that usually I mean historical not supported on gpus but tornado can support it and this is because they did compiler for example gpus don’t support dynamic memory allocation but sometimes in tornado we can do that because we compile runtime

[51:49] And we know more information about the actual applications so it kind of I’m not saying tornadoes are support a dynamic allocation and saying that because of the compilation nature of tornado because our runtime it can give the illusion to the program at times some things are supported on top of what opencell can leave here is extends to go compiler and the ground compiler is very well known for it’s a very aggressive and very good Escape analysis so all of those you cannot allocate an object in memory or GPU because gravel is very good at replacing options with scalars sometimes the developer believes that there is no object being allocated but because we use ground this object disappears and we still can run code that visually looks like that allocation but in reality it just has been replaced by scalars and sometimes it might get a little bit intimidating for developers how to transition the code from the pure Java to the xcpu tornado style of java requires some getting used to it we are working to improve our Tools in order to guideline developers towards that this kind of tools have been

[53:04] Existing on for let’s say CSC plus plus for many years like in television where you can actually get even advice on how to optimize it so we are working to provide something similar for tornado VM and forever but that wants to use acceleration filter interview so what are your advice to developers who want to take a look at their applications and just figure out where Tornado VM is a good candidate we welcome developers to try tornado in order to find out if tornado can help them they can just look if they have threading into their application if there is multi multi-processing or mass reading their application and they’re already using tablet address that’s a very good static point or indication that they can get high performance alternative try to analyze first application if it’s memory bound or compute bound if it’s compute bound and you need more compute power to process faster that’s a good candidate having enough threads to run so in rule of thumb don’t go to a GPU just to launch 10 threads so go to a GPU to launch hundreds or thousand threads right that’s another good candidate

[54:10] Another factor is also the data sizes because as we have seen it doesn’t make sense to go to a GPU for an array that has 16 elements because you will pay the overhead to transfer the data and this will penalize performance all right well thank you very much to all of the guests and thank you all for listening and please keep an eye on Foojay for future articles about development and everything related to the Java world