I don't claim to know anything but I thought tool usage was a major sign of intelligence. For example floats are a wonderful technology but people use them as if chainsaws are great for cutting bread and butter. We now have entire languages that cant do basic arithmetic. I thought it was alarming: People it cant compute like this! Now we have language models, those are still computers, why cant we just give them.. you know... calculators? Arguably the best thing their universe has to offer.
edit: I forgot my point: calculating big numbers is not a real world problem anyone has.
I'm not an expert at all, I learn they exist after making something similar to search a few thousand blog posts.
Rather than one hash per file I made a file for each 2 letter combinations like aa.raw, ab.raw, etc where each bit in the file represents a record. (bit 0 is file 0 etc) you could ofc do 3 or 4 letters too.
A query is split into 2 letter combinations. 1st + 2nd, 2nd + 3rd, etc the files are loaded, do a bitwise AND on the files.
a search for "bloom" would only load bl.raw, lo.raw, oo.raw, om.raw
The index is really hard to update but adding new records is easy. New records are first added as false positives until we have enough bits to push a byte to the end of each raw file.
I then got lost pondering what creative letter combinations would yield the best results. Things like xx.raw and an.raw are pretty useless. Words could be treated as if unique characters.
Characters (or other bytes) could be combined like s=z, k=c, x=y or i=e
Calculating which combination is best for the static data set was to hard for me. One could look at the ratio to see if it is worth having a letter combination.
But it works and loading a hand full of files or doing an AND is amazingly fast.
What you've described here is an n-gram inverted index (with n = 2) represented as a bitset. We could call it a bigram bitset inverted index. Glad to know you designed and implemented all of this from first principles, and that it serves you well!
I had a problem where we needed to compare large data sets between machines for keys that existed in both, and the bandwidth cost just wasn’t mathing for the median result set size. I was trying to figure out how to send a fingerprint from machine A to B, then have machine B send the hits back. Or how many round trips I could do based on set size to minimize bandwidth + latency. I ended up with a calculus problem nobody could help me solve because of an n^5 term.
My boss was generally pretty good with obscure data structures but neither of us had encountered Bloom filters. This was about three years after Google published their paper on how they were using Bloom filters, but that company would be bankrupt before I figured it out.
It doesn't matter what they think they want. If you give them more money everything gets more expensive. Employers should want the maximum productivity per buck spend. 8 hours, 5 days only has appeal to tradition going for it. I'm not aware of any serious research.
If you look at the revenue, number of employees and cost of living in the area it isn't so hard to calculate a sensible salary.
The weird thing in the west are these MBA types who feel they must force down labor cost even if it makes no difference for the company. I've seen lots of truly absurd examples of it. My favorite are the giant factories full of state of the art machinery and near perfect automation. 90+% of the employees are gone but the business logic still instructs to squeeze them. Like trying to squeeze wine from rocks.
If LLM's live up to optimistic speculation you can "soon" have a single employee run a large complicated software project with a low bus factor. Someone somewhere will think it is their job to make sure the dev costs the absolute minimum and works day and night. To spend $100 to squeeze $1 extra out of them is a job well done.
> Clearly, I cannot rely on having a Google account for production use cases. Google has built a complex, unreliable system
You cant use anything from Google. I only use gmail, my mail account only got banned one time for a week. For years I thought the punishment for using gmail was just a mater of time. I tried to imagine what weird things could trigger it. Maybe they will one day just end the service because it isn't profitable enough?
I decided the most likely would be that the mail account gets banned as a punishment for using any of their other services.
Then I made the "mistake" to switch from iphone to android. It almost immediately started complaining that my mailbox was full. The new reality is that each and every button I press on the phone could potentially end my mailbox.
Now that they [also] have very sophisticated LLM's the crappy customer service seems intentional.
I ponder this kind of things from time to time. This one makes a very enjoyable puzzle because it is extremely simply to move people on a conveyor or a rolling platform but amazingly complicated to get them on and off if you want to run it at any meaninfull speed, absurdly complicated.
My mental gymnastics is mostly trying to mobilize an entire city with the concept. I see some research suggesting there is lots of room between walking and driving distance. Bringing a bicycle also has its down sides.
Because getting on and off is already so difficult one tends not to notice the other problems with the tech. A simple crossroad is already a problem.
Moving fast is no issue for the more athletic passenger without luggage. It is also the most useless device to them. People with mobility issues don't have to get on but they do have to cross the road.
You would want to slow down or stop the surface, put a fence around it, you would want chairs, a roof would be nice, perhaps walls so that you can further control the climate. And then you have a bus, metro or tram. (haha)
One cool variation (not my idea) was to have a moving platform fromwhich to get in and out of a moving train, tram or metro. You could also make vehicles that connect on the sides. Those would have lots of fragile moving parts and potential dangerous situations if they fail. I see a night train misaligned with the station one time with the last door opening above the entrance of a pedestrian tunnel. A drunk guy almost walked out into the 5 meter drop. That seems preferable over falling between two moving trains.
reply