Foojay Podcast #56: Vectors in Java Code, Databases, and LLMs

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] In this Foojay podcast we enter the world of mathematics as we will talk about vectors and how they are crucial for AI and machine learning welcome to the Foojay podcast all your news about OpenJDK chat GPT explained to me that a vector is a mathematical structure that holds numerical values vectors are fundamental to the field of artificial intelligence as they allow mathematical operations to be performed efficiently and form the basis of machine learning algorithms okay but how are these factors crucial for the whole artificial intelligence Evolutions I want to understand and I have these amazing guests to answer all my questions I’m Jonathan Ellis I’m the founder of data Stacks so I worked on Apache Cassandra for a long time probably the most prominent Java based database out there and more more recently I started a project called J Vector which performs Vector search in Java and it’s used by Cassandra as well as data stx’s hosted Astra database so my name is Alexander Karas I’m a softare engineer at J driven so it’s a small consultancy in the Netherlands and I’ve been a

[1:20] Consultant now for eight years but I also really like to nerd out on new technologies and well llms AI that is one of those new technologies right now so yeah that’s me thank you thanks for joining this Foojay podcast I’m frankort technical writer Java programmer and also producer of this podcast and I want to learn more just like you said Alexander curious person so these vectors they’re crucial that’s something we already know is very important for artificial intelligence and machine learning but what is a vector can we start from the basics and can one of you explain to me what Vector is I think I can approach this from a high level and then we can kind of move down the stack a little bit from there basically when you’re when you’re talking to an llm and you’re you’re you’re sending it text or you’re sending it images like either way obviously you have to transform that into something that the llm can deal with and there’s a couple stages of that the first one is just tokenization of breaking that sequence of letters into a vocabulary that the tokenizer

[2:45] Knows about so I think I read that they typically have a I don’t remember what the vocabulary size is I’m sorry but so I’ll refuse the temptation to guess but these are often open source so even open ai’s tokenizer is open source on GitHub I believe it’s called tick token and so there’s that’s considered to be more or less a solved problem like we know how to tokenize English or German or other languages into this smaller vocabulary which are pieces of words it’s roughly the guideline open AI gives is you have roughly three4 of a word per token on average and then the next step is to encode those that vocabulary into or that tokenized sequence into something that you can feed into the language model into the neural network and so that’s the first place where we turn that into a into numbers right into numbers and then internally inside the LM you have these different stages of representing what we fed into it and each layer within that neural network is a fixed size and so what you can do with these Transformer models is

[4:14] You can take one of those layers and just say I’m just going to rip out that sequence of float 32s that’s in that layer and that sequence represents the state of the model at that layer and so it’s deterministic you can feed it forward through the network and given the same sequence you’ll get the same output and so one of the things that you can do with that is you can represent this you can take an arbitrary sequence of language and then you get out of that by feeding it into the network and pulling it out at one of those stages one of those layers you get a fixed size representation of that language and so when you see people talking about encoding models that’s basically what’s that’s basically what they’re talking about that they’ve they’ve gone from something that can take you know it’s it’s not an unlimited sequence but a long sequence of text and turn it into a fixed siiz vector and when you train an encoding model you’re trying to build it such that similar Concepts result in similar vectors or vectors that are close to each other and

[5:36] Typically you’re using cosign distance to measure the closeness of those vectors and so early like one of the earliest of these was Google they published a paper Pap yeah 2013 the per paper wasn’t called word Tove but the project like the code was word Tove and so they came out with some really interesting results like if you take the vector for King and subtract the vector for man and add the vector for woman then you get something very close to the vector for Queen right and so and so there was a lot of emergent properties that people started getting fascinated about this word to V like the name suggests was just about single words but then they started being able to expand that to where we are today where open AI will sell you a service that will tokenize to 1536 Dimension vectors or the large one now is twice that size cohere has a service that does something similar Nvidia has something similar anthropic has something similar there’s a bunch of open- source ones so yeah it’s it this is and so I mentioned earlier that the tokenization is not

[6:57] Considered a place where people have like competitive secrets and open ai’s got theirs open source but the actual encoding like that’s that’s very much like everyone’s got their own secret sauce and it’s very hotly competitive right now so these factors are a series of floats and very easy to search to find the closest relatives Alexander at Fox days Brussels I saw you gave a presentation about this subject you took a very strange approach of illustrating this with a game and finding the matching card or a matching action of a game can you explain how this helps to explain how vectors are used well well the thing is I have my background in game so whenever I try to learn something new I try to make a game out of it gamify it a little bit I guess because I’m a visual learner so then I thought okay I play Dungeons and Dragons so is it possible for me to just find spells in Dungeons and Dragons but just querying random things and try to find what I did what I’m searching for and that but the big difference that I between the demos I used and what

[8:17] Jonathan just said is I didn’t use a tokenizer in between so typically you want to use a token tokenizer to you know or text splitter a word splitter or a graph splitter to make slow smaller chunks and then tokenize as words and then vectorize everything and I just send the whole text as it was to an embedding model and got a vector representation then saw what I could find and what I couldn’t find so it was like a couple steps before like the whole llm stack so to say it was really pure okay I have two Vector representations one what can I type to find what I search for or do I need to change my data models do I need to change my data ingestion pipelines you know stuff like that so it was very much more experimental and being able to visualize it in game made it for me very easy to you know trial and error okay this doesn’t work this works I continue on from that I found a recording of your talk of devox I think I will add it to the show notes for people interesting in this I find it I find it very informative of how you try to explain these complex

[9:35] Things of factors and how you can find the nearest match of them with this game approach so if I go back to the tokenizers so a tokenizer is that really tied to how it’s implemented so the open AI tokenizer could you just use it for another type of application or is it really tied to how their chat GPT works I work with vectors like once I get the vector right so somebody else is building the encoding model and the tokenizer and everything else so I mostly work with it like here’s a here’s a billion vectors and I want to be able to find the closest matches but my understanding is that the tokenizers are not particularly tied to a single model and you could potentially reuse that in another project yeah I think so as well from what I’ve understood and read I think it’s pretty reusable left and right then we also have the topic of databases so I come from a generation with transactional databases is an Vector database completely different because you only store numbers or can you do can you do it with traditional database too this is where this is where I have a strong

[11:04] Opinion that is potentially controversial but I think I’m going to be proved correct in the next couple years which is that it Vector search is useful but you don’t need a special Vector database to do useful Vector search and so if you look at a you know a ve you know a Vector database like pine cone like the vector is your the vector is always your primary key to use database terminology and so there’s no such thing as a query that doesn’t involve the vector in other words so for instance let’s say I have I want to be able to track the articles that Jonathan Ellis wrote and so with a vector database I can query for show me the most similar article or show me the most relevant article that Jonathan has written about Java strings like I don’t know some feature of java and I can do that with the vector database because that’s a semantic search but I cannot ask it for show me the five most recent articles that Jonathan wrote like there’s no way there’s no way for it to do that because that doesn’t involve that Vector and so the approach

[12:32] That we’re taking with Astra and I think and with things like PG Vector as well is that you want to be able to do both and you want to be able to do them both with the same database rather than doing those with two different systems and am I right so that databases like Maria de or postres they have plugins to help the use of vectors yeah so Alexander probably knows more about this than I do but I know that post gql famously has PG Vector I’m not aware of one on the MySQL side but it’s probably due to my ignorance there same here but yeah that’s and I think all the big database vendors by now also support some form of vectorization mongod Tob has it in afg you can vectorize your notes in the graph database which is kind of interesting to me so yeah they all try to support it at least by now I don’t have this much knowledge about this topic so I’m I’m really struggling to find how so if a vector database is only numbers and floats How does it go back to a chat answer is that the model that’s in between which uses this to build up an answer so the vector database doesn’t

[14:02] Contain any text anymore or does it can for example in the demos I use my presentation so like Jonathan said the vector is literally the primary key but when I hit something out have all kinds of associate metadata with this object with this Vector and then I will just retrieve that so then I know all kinds of properties about what I just found and then can use that however I want but I’ll be honest I’m I don’t know I’m not sure if that is exactly also how llms work with a vector search I don’t think so but I’m not sure the vector search is used as part of retrieval augmented generation or rag so that you typically have your llm you want to give it some you know llms are notorious for hallucinating and making things up when they don’t know the answer and so one of the best ways to ground it in reality is to give it some background information about the question you’re answering and so that’s where the vector search comes in is you take the query that you’re trying to give to the llm and you go to your database of context and you say what are the most relevant

[15:23] Documents to this query and then you pull those out and give it to the llm as part of its as part of your prompt that you’re giving it that’s something I’ve been experimenting myself to search in the aul documentation without promoting a python library for instance and that’s what I ended up B my first test is I want my system my chat system to only answer Java related questions so that’s a bit what you do with is rack so you give it yeah a list of topics and ask it politely to stay within those topics and but then still that hallucinating can happen correct yeah it’s actually this is actually a place where something that looks straightforward is actually fairly difficult because what Vector similarity can give you I’ll back up for a second so you mentioned early earlier that you know hey this is an easy thing because now we have two vectors we can compute the distance and that gives us a the nice thing about a cosine distance is it takes however many dimensions your vectors are and brings that distance down to a single float right and so that’s I

[16:43] Can no M even if I’ve got 1536 dimensions and I’ve got three vectors I and I can get one float that’s saying even though on Dimension 12 and 13 I’m very different I’m close enough on the other thousand Dimensions that this ends up being closer but what’s difficult to do is to set a threshold and say Above This threshold is relevant Above This threshold is related to Java and below this threshold it’s not because you just have this continuous you know minus one to positive one cosine distance and the models at least today aren’t aren’t trained to be able to say you know o you know compute vectors such that above a certain cosine these are close and below it they’re not so often what you’ll end up with is and this varies from model to model from encoding model to encoding model with BG BGE one of the open source models for a Wikipedia search the demo that I was doing relevant was a above about 7 on the cosine distance and using open AI relevant was above uh85 something like that and it’s very it’s very squishy so yeah you can try to do things like that but that does get

[18:15] Into the territory of it’s actually more difficult than it looks unfortunately ra rment generation is all about giving context right so that I really like what Stefan said Dre at Brussels at this talk about ra that as the llms become bigger and accept more tokens like Gemini accepting one million tokens or something right now then the need to vectorize your own data and put it in a well Vector store or vector database or whatever you want it is GNA diminish so see that’s something that I strongly disagree with because two for two reasons first is kind of a technical limitation which is that as you start to as as you expand that context and you dump more and more data into it the llm still has trouble kind of paying attention to everything at once and it does okay with when people do needle in Hast stack searches right so they give it you know here’s a million tokens Now find the part where I asked well where I said what Paul Graham had for lunch right and I’ll I’ll sneak something into this document that has nothing to do with what Paul Graham had

[19:40] For lunch and I’ll I’ll I’ll have it look for that needle in the hay stack they’re actually pretty good at that on balance but when you start asking for multiple needles then the performance gets not nearly as good so there’s some there’s some technical issues there just in terms of being able to track multiple things across a very large context but the bigger problem is that the more con if you giving giving an llm irrelevant context is worse than not giving it context at all and so there’s you know you still need to prune what you’re giving in llm and make sure that what you’re giving it is actually relevant to the question you’re asking or you’re you’re going to be in a case of garbage in garbage out it you can’t just throw everything at it and hope for the llm to sort it out it’s not good at doing that Alexander you were working on a few projects and proof of concepts of using chat interfaces you were talking about like finding two locations in a question do you are there already many use cases in the field being actually used at customers that you can talk

[20:59] About where they use such a chat interface being a consultant that do tend to talk to some companies at least in the Netherlands I cannot speak of the United States or anywhere else I don’t know exactly what’s happening it’s probably much different than here in the Netherlands but here after there was quite some a lot of companies that wanted to try chatbots with llms but after what happened with Canada air where the chatbot was like yeah you can have your money back and then they had to actually do that a lot of them backed out immediately like there it’s not it’s not they were like yeah it’s not mature enough it needs more work etc etc etc so yeah we’ve been mostly experimenting to see if we can use Vector search and more interestingly multilingual embedding models to allow users from all over the world world to search for certain cities in our application I work for root scanner.com which is like a door-to-door container Logistics application so yeah and everybody hurg is written different differently in German then it’s in English then it’s in Chinese so that is

[22:17] Very interesting to us if we can use that instead of having an elk or an elastic application server somewhere with all the multilingual tags etc etc MH but a real application like an RA llm application we have not found yet or I’ve not seen yet better said it’s it’s it’s nice to see how much companies are researching this topic but are a bit hesitating to actually put it in production or definitely put it into something which is publicly facing because then you end up yeah with these lawsuits and things being promised which cannot happen I think that that’s definitely true that it’s it’s much easier to build a very compelling demo with an llm than it is to turn that demo into a product so actually one of separate separate from what I work on at data Stacks we have a team building something called rag stack which is an open- Source Library that incorporates kind of the best practices around productionizing this kind of thing so we I’d be happy to introduce you to one of our Engineers named Christoff bour who would be able to go much deeper on that a perfect topic for the

[23:43] Next podcast Jonathan on Foojay you published a popular post about indexing all of Wikipedia on a laptop what was the goal of this blog post demonstration of what can be done now with the vector database in J Vector my goal with that article was to Showcase I guess on two axes how far Vector search has come specifically Vector search for Java that you know with J Vector we’ve built a library that can do things that nobody else can do and I wanted to showcase that by indexing one of the largest public data sets on my laptop and so that’s what that’s what I set out to do and you know basically with a 40 40 gigabyte Java Heap that that’s enough to index all of Wikipedia and serving the searches once the index is built only requires a 4 gigabyte Heap so you know both of those both of those tasks building the index and then serving it with a very small footprint those those are things that nobody’s done before so if I read correctly that data set is 180 gigabytes correct right and so that’s that’s both the vector data as well as the

[25:18] Associated text and URLs and so forth yes so the entire data sets about 180 gigabytes and that gets converted to vectors now you have to admit it’s a decent laptop with 40 G of memory yes it’s a it’s a 64 gab machine yeah but it still means that after this huge task is done once you can run it with 4 gigabytes of memory correct so I can put it on my Raspberry Pi and serve it from website because your demo is actually available online people can search through it goes very fast so I don’t know on what kind of machine is it running the demo you know I’m actually not sure one of my colleagues at data Stacks threw it up on an Amazon VM but I don’t I don’t think it’s a very large one and what were the challenges or is it just if you follow the blockpost I can do this myself and I don’t need very detailed knowledge of vectors in the database and how this whole process works I mean I ran through the steps two or three times to make sure that worked as advertised so I think it will just work certainly if you want to go deeper and start customizing it and

[26:35] Figure out like what what’s going on under the hood then you’re going to need to dig into some code but it should just work and I haven’t tested it on a Raspberry Pi but I imagine so J Vector runs on everything down to jdk11 but if you’re running on JDK 20 plus then we can take advantage of the SIM from Project Panama so that gives it approximately a 2X performance boost I don’t think that we I don’t think there’s any simd support on Raspberry pies I could be mistaken so I wouldn’t expect it to be super fast but I would expect it to work I’ll put it on my to-do list I have the raspberry pies behind me but what I’m missing is a bit of time but you mentioned a good topic so from what was it vers 21 22 of java so actually from 20 the subset that J Vector uses is the same from 20 and later and so they’ve added new things but they haven’t broken anything that was released in Java 20 Java 19 is too old things did change across there but from java 20 and on it works and I’ll I’ll cave at that by saying that there’s actually some new stuff in

[28:03] 22 specifically around the native memory representation where you can in older versions if you wanted to take a you know a float array and do stuff with native code and then bring that back into the JVM you actually had to copy the JVM float array into a native memory area which obviously you know copies are the enemy of performance and in 22 you can actually just wrap that J you know internal JVM float array as a memory area that native code can access with high performance and so we actually we take advantage of that for some of the optimizations but the basic Vector search simd acceleration doesn’t require that we have the six- month release cycle with improvements in every version how much improvements are you expecting from the next release so you already mentioned forign function and memory API from java 22 was JEP 454 for Java 23 which will be released in September we have the vector API but it’s already the E incubator so this means that they already were Evolutions in the eight previous editions of java releases and it’s still not

[29:32] Finished this Vector API is this something completely related to the topic we are discussing now or will it offer more or different functions right so there’s a bit of a terminology Collision here because obviously when we’re talk when we’re talking both both both uses of the word Vector are dealing with sequences of num num but in the JDK when you’re talking about you know the vector API what we’re really talking about is simd of being able to use Intel AVX and arm neon and that kind of hardware for applying a single logical instruction to a large area of memory and so the overlap is that when I have you know these mathematical vectors or these llm vectors and I want to compute the cosine similarity like that that’s actually very well suited for simd hardware and so I want to use the JDK vector API to do that work and will this improve the things like they are now a lot yes so basically we’re I’m I’m really happy with where it is right now so we can do almost everything we want inside inside the JVM but there’s a there’s a footnote there next to that

[31:06] Almost everything where I mentioned earlier that for some things we do need to go out to native code because there’s no way to express it performant using the JDK vector API so some of those I do think some of those limitations are going to go away but other limitations it’s just like they’re always going to be with us and we’re always going to need native code for some pieces but that’s the other cool thing that’s happening in Java recently is the foreign function api’s gotten much much better so that’s another thing we’re really happy happy about and yeah we’re I I’m I’m a huge fan of you know back back when Oracle acquired Sun a long time ago people were like oh no what is this going to mean for Java is Oracle going to screw it all up but h hly I can’t complain about Oracle stewardship at all like and not just Oracle obviously everyone else contributing to OpenJDK you know red hat and aul and so forth really delighted with the progress that’s that’s happened there for high performance Java applications and what most people Al also say about this is they’re happy

[32:23] With evolutions and happy with the stability because these evolutions they do happen but they don’t break what was already there or at least once you get once you get from jdk8 to jdk9 like that’s the that’s the big tough move once you get past that it’s pretty smooth sailing so I have a question maybe for Jonathan if that’s okay do you I mean since you work on like a library like J Vector do you miss the ability to work with the GPU directly or not or do you do that through the foreign function API so I actually believe that the GPU is the wrong tool for Vector search almost always because what the GPU does is it gives you a very limited amount of memory that you can do very fast operations against and so you know for my entire career in databases there’s been a small group of people saying Ram is all you need like just build a database that does everything in Ram checkpoint to disk for durability but you don’t need disk for large data sets just spread it across more machines in Ram and that’s never worked out generally speaking in the industry because we’ve been able to generate

[33:48] Larger data sets as fast as we’ve been able to create denser denser dram and so you just have much better cost performance if you follow the parito principle and you have you know your warm data on disk and your or on SSD now and your hot data in Ram but to try to keep everything in Ram it’s just a foolish use of money it’s not cost effective and the GPU is like the next step of that it’s even more true for the GPU because a big GPU has 80 gigabytes of RAM compared to a big server that has 2 terabytes of RAM and if two terabytes of ram isn’t enough isn’t cost effective then trying to do that across 80 80 gigabyte gpus is even less cost effective and those are expensive those are really expensive right that’s true so yeah you can get blisteringly fast Vector search if you limit your data set to what fits in a GPU and Nvidia has a library to do that and that there’s some there’s some projects that embed that Library out there but I just don’t think it’s a very useful thing to do from a cost benefit analysis the vector API is so it’s still an incubator it takes some

[35:09] Time do you know that people are really waiting for it they want to use this they need it for certain use cases certain applications I mean we’re using it in production right like all of Astro deployers with the vector API enabled and like you said it’s it’s still incubating we’re on version number eight but it’s it’s stabilizing right like so from I mentioned that since JDK 20 you know we run the same code on 20 2122 even though that’s three different versions of the vector API so there’s there’s changes but they mostly changes around adding things now and saying oh this is going to be useful let’s put this in rather than saying oh like we screwed this up and let’s remove it and break people’s code so I mean it could happen like it could happen the JDK 23 or 24 says we’re going to change things up and you need to rewrite your code if that happens we can adapt like all of the all of the vector API code is self-contained and we’d be able to adapt that if necessary so I would say don’t be afraid of the incubating label especially once it’s gone through that first couple

[36:25] Iterations obviously there’s there’s different appetites for that kind of risk depending on your project and so forth Alexander is that something that you seen production that that’s people tend to use incubator features probably not no not really not not the kind of companies I typically work for like the big government institutions they tend to shy away from anything that is experimental incubator etc etc etc and do they stick to the long-term supported versions yes always so we typically never go from well now Ro can we do but like Port of Rod Dam other big government institutes typically they go from LTS to LTS never in between so they never really upgrade in between but it is a way this six-month release cycle for developers to really experiment with what’s happening in Java and be prepared that as soon as something becomes a final ver verion in a long-term supported release that they can start using it I mean arguably but it really differs per team right I’ve heard I’ve heard it often that they say yeah but we’re not allowed to use that

[37:43] Version so why even bother experimenting with it in our local Stacks or whatever and things like that so it really diff that’s why it’s such a good balance though right because this LTS every two years and the six-month releases in between means that the very conservative organizations can go from 17 to 21 to 25 and that’s fine but then companies like data Stacks can start using those you know improved foreign memory access apis in 22 as soon as it’s released and not have to wait for that extra two years so I I’m really happy with the balance and I think it does a good job if you look at the future of chats and these large language models what are you expecting I know this is very hard question in the next year because things change so fast that we could not imagine two years ago where we are now I guess I’ll give you a prediction right right now almost everything is being done with single Vector representations and so U you know Alexander mentioned mention that part of the state-of-the-art approach is if you have a larger article you do want to break that up into

[39:04] Chunks and you know those chunks are often going to be sentence length or you know two or three sentences but the larger you make them the less specific the content gets and so that comes back to what I was talking about earlier is you do want to you do want to give the data the model only the most relevant data and so you don’t want to give it like entire documents worth you want to give it paragraphs Worth or subparagraphs but the state-ofthe-art right now is to take that paragraph and convert it into a single vector and then the next paragraph into a single Vector the problem with that is that despite the best efforts of the embedding models developers it it’s difficult to perhaps impossible to capture the full semantics of a you know 200 or so word chunk of text in a single Vector even though it’s very large Dimension it’s just really difficult and to be more specific what we see is that embedding models struggle with outof bounds data and what I mean by outof bounds is a tokens that are not often present in the training data and so one of the specific areas that this

[40:29] Comes up with is people’s names and so you know looking for if if I just go to a vector search and I type in Jonathan Ellis or I type in Frank delport I’m probably not going to get very good answers out of that because it wasn’t you know there weren’t a lot of Jonathan Ellis’s in the training data and there weren’t a lot of Franks in the training data and so what one research out of Stanford who he just said he got a tenure track position at a university somewhere else I don’t remember where but he did his he did this as part of his PhD work at Stanford he created something called Colbert and what that is it’s a way of instead of creating one vector per chunk you create one vector per token and that Vector per token is uses a Transformer model to semantically shade or semantically adjust the vector that comes out for each of those tokens and so then so now instead of having one vector for that 200 words you have 200 vectors for that 200 words and then your search becomes I want to capture the aggregate similarity across multiple

[41:55] Vectors it’s actually a more computationally challenging problem as well as it gives you much better results much more relevant results because it solves that problem of outof bounds data be by working at the token level so I think that’s I think that’s the future of vector search I would say that probably only the most Cutting Edge Vector search nerds know about it right now but I think it’s going to be a lot more relevant in the next 12 to 18 months will it also have an impact on the cost ecological cost energy cost because I don’t know I’m not sure Alexander if you shared that slide or it was in another presentation there is this graph where you see the cost the energy cost of doing a search difference based on difference of the model if you say that instead of one vector you will have 200 vectors I can imagine that the storage will be higher the number of calculations to find these will be higher to the first part of the question there’s active it’s it’s a very much an active area of research to be able to figure out how do you compress this

[43:18] Better how do you narrow down your scope of search better using those multiple vectors because part of it you can so the first cobber paper said hey let’s just throw a standard Vector index at this and that worked but it wasn’t very fast and it took a ton of storage and so then the next papers that he wrote building on that were let’s let’s figure out how to do this more efficiently and I don’t think we’ve we’ve hit the limits of that I think there’s more to discover there but for the second part I mean this is where I’m I’m I’m in I’m on team nuclear I definitely think that we should be looking we should be spending a lot more effort on how do you how do you make clean energy cheap and ubiquitous rather than trying to figure out how do we make people use less energy which is a big challenge for the future I guess something else that you want to talk about or that people need to know about the whole Vector thing I will I’ll I’ll throw out one more thing which is that at the so this is a little bit more in the weeds than we’ve talked about

[44:38] Most of the show but I’m I’m going to go there one one way to make Vector search less expensive and more efficient is to make your vector smaller and so that’s called quantization like how do I capture most of the information in this 1536 Dimension Vector in a much smaller compressed version and there’s a there’s a couple ways of doing that the most effective in general is called Product quantization and the most buzzwordy is called binary quantization and the reason that I draw the distinction there is that I think product quantization is almost always is better but binary quantization is much faster and so that’s why it kind of makes headlines because in binary quantization you take your float 32 your sequence some float 32s and if each if a float 32 is negative you encode it as zero if it’s positive you encode it as one that’s that’s binary quantization and so now to compute the similarity between those compressed vectors it’s just a Hamming distance it’s just how many of those zeros and ones are different in each position across the vector and you can do that

[46:00] Like blisteringly quickly on Modern Hardware like there’s there’s there’s you know assembly level Primitives to say count the different bits in this you know 64-bit long the problem or the reason why I like product quantization even more is that you lose less accuracy and you can compress things even more which helps your cost model so binary quantization obviously it’s a fixed 32x compression when you’re starting with float 32s with product quantization I can actually compress 64x with the popular model so I can get better compression and almost as good speed the thing that I wanted to call out for the listeners here is to make sure if you’re using a vector database or a vector search engine that supports compression to make sure that it also supports reranking the compressed results using the full vectors because you lose a lot if you just do a search using the compressed versions you will lose half of your accuracy like a lot of accuracy and so the way that you get by that is you search deeper you search twice as deep or three times as deep as

[47:18] You normally would and then you rerank just those top results using the full Precision vectors and so that’s something that a lot of a lot of products have or and projects have added the first part but not the second part and so if if you don’t understand the distinction then you might be wondering like why is my accuracy terrible I don’t understand what happened and it’s because you’re you’re just throwing away a lot of information during the compression with no way to recover that unless you do the reranking okay Alexander correct that in your presentation you also explain this binary approach no but I did use future projection so that’s also a way to smaller the amount of vectors you have maybe you remember I like this demo was flying through space and I showed that semantically vectors that are semantically close to each other they end up close to each other and then I brought vectors that are 300 384 Dimensions back to three so I could plot them in a 3D space and that is why did use such things or of course or to make your vector search faster people people struggling with the

[48:35] Concept of the vectors and how they are what the distance is between them I really like the 3D model presentation of that you have these clouds of topics which are very close to each other and that’s very nice visualized in such a space Vector space a vector space and actually it’s a perfect thing to create games Sean Phillips who also who creates a Java vix application to visualize data like that he actually created and published something on Twitter last week with a space invader game flying through his data so if you want another another example of com combining games and data that’s the place to be okay thanks a lot for your time thank you learned a lot again H thanks for explaining this very technical subject to me and the listeners of the Foojay podcast so thank you also to the listeners H keep an eye on Foojay for future articles and podcasts about development and everything related to the Java World thanks a lot give me a give me a j give me the friends of OpenJDK