```
We evaluated several precision pairings across our internal retrieval benchmark suite. Scores are NDCG@10 averaged across the suite, scaled to 0–100. NDCG@10 (Normalized Discounted Cumulative Gain at rank 10) measures how well the top 10 results are ordered against the ideal ranking, rewarding relevant documents more when they appear higher, with 100 being a perfect ranking. The full-precision baseline averages 90.26. Int8 query against binary documents averages 89.65, a 0.61 point drop, while reducing document-vector storage by 32x
```
Saying "Near lossless" to mean 90% accurate retrieval of saved vectors is simply a lie. Lossy-ness is binary, not something you can paper over with getting close enough. And 90% is not close. Sure, LLMs are all about gradient descent on noisy data sets so I guess this is acceptable in this field but that terminology usage still bothered me
kittoes 2 hours ago [-]
I don't believe that's what they were saying at all though. The claim appears to be that it's near lossless relative to their own baseline that uses float. Which I'd grant, since a 32x storage reduction for 0.61% loss in quality is a reasonable trade off when you've already decided to accept that ~90% is "good enough".
seritools 1 hours ago [-]
near lossless refers to being 89.65/90.26 = 99.32% of baseline, i'm pretty sure.
breadislove 49 minutes ago [-]
yes exactly.
elil17 5 hours ago [-]
I would love to see real examples of what reduced quality means in practice. Are you able to recover a document from the vector in a human readable format? If so, what sort of changes come up?
I could imagine a scenario where differences tend to be more substantive than you'd expect because of how less frequent words with fine distinctions in meaning - the very words that make the document special - may be embedded in the vector space.
yorwba 4 hours ago [-]
Most of the fine distinctions are already lost when a document is processed through a pile of linear algebra to turn it into a fixed-size list of floating-point numbers, as you can see from the NDCG@10. Vector search is not a tool for fine distinctions. It's a tool for reducing a large pile of documents to a smaller selection of candidates, which you can then check individually with some more expensive method.
breadislove 40 minutes ago [-]
The ndcg loss is minimal 90.26 -> 89.65. This means it maintains most of the quality.
breadislove 48 minutes ago [-]
this is the reason why we report ndcg and not recall. ndcg respects fine grained details so you get the an overview of how much details you are trading off since it would hurt the ranking.
purple-leafy 5 hours ago [-]
Hey breadislove; amazing article, I’ll be sending mixedbread an email in the morning that may interest you (email will be <5-characters>@pm.me)
I have also been working in compression and performance engineering, and managed to get a 99+% compression unlock versus conventional approaches (100+KB down to 1KB) in the scenario of 30 minute massive multiplayer game replays for a “game+engine” I’m developing
I think there’s a synergy between these 2 concepts I’d love to chat some more
breadislove 47 minutes ago [-]
to which email did you send it? can u send it to support please?
palinnilap 2 hours ago [-]
Any way I can read about this or the use case? I have a hobby interest
5 hours ago [-]
kaizenite 2 hours ago [-]
To people smarter than me, how impressive and/or revolutionary is this?
Ameo 8 hours ago [-]
I can't wait until we get to 100% storage/cost/compute reduction for LLMs. Every thought you could have thought pre-conceived in high-fidelity super-resolution. Every action you could have taken predicted and simulated in advance courtesy of Openthropic and the USA Sovereign Wealth Fund.
mwigdahl 56 minutes ago [-]
Unfortunately as cost reduction trends to 100%, it comes along with an intrinsic high-pass sarcasm filter.
throwaway2027 4 hours ago [-]
You would obviously be trading storage for compute and time to retrieve the storage.
throwaw12 7 hours ago [-]
100% reduction is impossible for something which should work, because -100% means it is now 0
neonstatic 6 hours ago [-]
They were clearly being sarcastic
peheje 7 hours ago [-]
Reminds me of 'Learning to be me' by Greg Egan
alfiedotwtf 3 hours ago [-]
If you squint hard enough, it sounds like their storage layer is a bloom filter
functionmouse 5 hours ago [-]
there is no such thing as "near lossless"
ttoinou 5 hours ago [-]
There is, after you define what you’re ready to loose and understand the lossy space. That’s how we came up with mobile cellphones, audio and video codecs etc. Literally powering all modern devices we use.
greenleafone7 2 hours ago [-]
So then ... "lossy"
2 hours ago [-]
functionmouse 3 hours ago [-]
Actually, all of those things are considered "lossy".
ttoinou 2 hours ago [-]
Yes, anything not lossless is lossy. Near-lossless is not lossless, so it is lossy. I hope we speak the same language
rq1 6 hours ago [-]
The Pi compression algorithm is better.
luma 3 hours ago [-]
Doubtful. The problem with the pi idea is that you need to include the offset, which will likely be as long as or longer than your data.
nathan_compton 3 hours ago [-]
" A single document produces more then one embedding, depending on the complexity of the document it can produce hundreds or thousands of vectors."
That typo up there is kind of endearing in the AI slop era.
HenryMulligan 1 hours ago [-]
Not seeing a typo in your quote. Can you point it out?
thatspartan 34 minutes ago [-]
I think they're referring to "then" vs "than"
breadislove 23 minutes ago [-]
ah whoops, I'll fix it. ty!
m_m_carvalho 4 hours ago [-]
[dead]
mv_d5339e31 7 hours ago [-]
[dead]
johnathan101 8 hours ago [-]
[flagged]
7 hours ago [-]
TradingReality 2 days ago [-]
[flagged]
Rendered at 16:24:31 GMT+0000 (Coordinated Universal Time) with Vercel.
Saying "Near lossless" to mean 90% accurate retrieval of saved vectors is simply a lie. Lossy-ness is binary, not something you can paper over with getting close enough. And 90% is not close. Sure, LLMs are all about gradient descent on noisy data sets so I guess this is acceptable in this field but that terminology usage still bothered me
I could imagine a scenario where differences tend to be more substantive than you'd expect because of how less frequent words with fine distinctions in meaning - the very words that make the document special - may be embedded in the vector space.
I have also been working in compression and performance engineering, and managed to get a 99+% compression unlock versus conventional approaches (100+KB down to 1KB) in the scenario of 30 minute massive multiplayer game replays for a “game+engine” I’m developing
I think there’s a synergy between these 2 concepts I’d love to chat some more
That typo up there is kind of endearing in the AI slop era.