So heat. There’s efforts to switch to optics which don’t have that heat problem so much but have the problem that it’s really hard to build an optical transistor. + anywhere your interfacing with the electrical world you’re back to the heat problem.
Maybe reversible computing will help unlock several more orders of magnitude of growth.
You’re comparing a 4 year bloodbath to 10 minutes and being underimpressed? Also those weapons are several orders of magnitude less powerful than what they’re capable of today…
Battle of Carthage was also 3 years and was a siege of a city, so you know… not a lot of places for the people inside to escape. Also took about 20-50k expertly trained Roman soldiers vs a few trained guys in a plane pressing a button.
And sibling comment is right. The application of industrialization to the death process in WW2 and similar application of the idea (eg Pol Pot and Stalin) also led to death on an unprecedented scale.
That caused endless tragedy and trauma. Perhaps the 10 mins terror was the less worse outcome of the two, mode decisive, that ended the war quicker. Who can decide? Wars aren't statistic.
That’s a weird tldr and not my takeaway. More like “scientists convinced their new ultra destructive weapon is sure to bring about peace this time around”. Spoiler: it does not. Arguably maybe nuclear weapons but even then I’d say the use of nuclear weapons in armed conflict hasn’t really been tested yet and people are generally hesitant to do so, preferring instead illegal chemical and biological warfare.
Even completely ignoring the issues of language-centric vs data-format-centric serializers, your list is missing two very notable entries from my list: Arrow and Parquet. Both of them go to quite some lengths to efficiently handle optional/missing data efficiently. (I haven’t personally used either one for large data sets, but I have played with them. I think you’ll find that Arrow IPC / Feather (why can’t they just pick one name?) has excellent performance for the actual serialization and deserialization part as long as you do several rows at a time, but Parquet might win for table scans depending on the underlying storage medium.). Both of them are, quite specifically, the result of years of research into storing longish arrays of wide structures with potentially complex shapes and lots of missing data efficiently. (Logical arrays. They’re really struct-of-arrays formats, and I personally have a use case I kind of want to use Feather for except that Feather is not well tuned for emitting one row at a time.)
I have zero doubt that you’re on some ‘no true Scotsman’-style “you’re not doing Real Development if you are using these technologies to solve these problems” thing. Let’s just drop that. There are myriad ‘real man webscale development’ scenarios where these are more than acceptable.
Pretty sure protobuf used a header to track field presence within a message, similarly to what this article does. That does have its own overhead you could avoid if you knew all fields were present, but that's not the assumption it makes.
Actually, it's you who is giving that impression with an ultra vague "doesn't solve the problems described".
The only problem in the blog post is efficient coding of optional fields and all they was introduce a bitmap. From that perspective, JSON and XML solve the optional fields problem to perfection, since an absent field costs exactly nothing.
I guess you missed the part where the size of the data stored on disk and efficient deserialization are also critically important performance characteristics that neither JSON nor XML have?
Capnproto doesn’t support transform on serialize - the optional fields still take up disk space unless you use the packed representation which has some performance drawbacks. Also the generated capnproto rust code is quite heavy on compile times which is probably some consideration that’s important for compiling queries.
> Protobufs definitely doesn’t solve the problems described. Capnproto may solve it but I’m not 100% sure. JSON/XML/ASN.1 definitely don’t.
I'm not sure you are serious. What open problem do you have in mind? Support for persisting and deserializing optional fields? Mapping across data types? I mean, some JSON deserializers support deserializing sparse objects even to dictionaries. In .NET you can even deserialize random JSON objects to a dynamic type.
Can you be a little more specific about your assertion?
The space overhead and the overhead of serialization/deserialization. Rkyv is zero overhead - it’s random access without needing to deserialize and can even be memory mapped.
The whole “zero overhead” thing is IMO a red herring. I care about a few things: stability across versions and languages, space efficiency (sometimes) and performance. I do not care about “overhead” — performance trumps overhead every time.
Your deserializer is probably running on a CPU, and that CPU probably has a very fast L1 cache and might be targeted by a compiler that can do scalar replacement of aggregates and such. A non-zero-overhead deserializer can run very quickly and result in the output being streamed efficiently from its source and ending up hot in L1 in a useful format. A zero-overhead deserializer might do messy reads in a bad order without streaming hints and run much slower.
And then to get very very large records, as in the OP, where getting a good on-disk layout may require thought. And, frequently, the right layout isn’t even array-of-structs, which is why there are so many tools designed to query column stores like Parquet efficiently.
Serdes time can be significant. There are use cases for the zero copy formats even though they use more space. Likewise bit-packed asn1 is often slower than byte-aligned.
If you care about space, you're almost certainly going to compress your output (unless, like, you're literally storing random noise) and so you'll necessarily have overhead from that.
Unless the reason you care about space is because it's some sort of wire protocol for a slow network (like LoRaWAN or Iridium packets or a binary UART protocol), where compression probably doesn't make sense because the compression overhead is too large. But even here, just defining the data layout makes sense, I think.
Tihs could take the form of a C struct with __attribute__((packed)) but that is fragile if you care about more platforms than one. (I generally don't, so that works for me!).
Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set accidentally due to the closeness of & and && or | and ||.
Deduplicating and identifying the source of a crash point is surprisingly hard, to the point that “it’s the only crash of its kind” could be a bug in your logic for linking issues.
Also, in an unsafe language all bets are off. A memory clobber, UAF or race condition can generate quite strange and ephemeral crashes. Even if the majority of time it generates the “same” failure mode, it can still sporadically generate a rare execution trace. It’s best to stop thinking of these as deterministic processes and more as a distribution of possible outcomes.
Deduplicating and identifying the source of a crash point is surprisingly hard, to the point that “it’s the only crash of its kind” could be a bug in your logic for linking issues.
This is a bit vague to really reply to very specifically, but yes, this is hard. Which is why quite some people work in this area. It's rather valuable to do so at Firefox-scale.
Even if the majority of time it generates the “same” failure mode, it can still sporadically generate a rare execution trace.
This doesn't matter that much because the "same" failure mode already allows you to see the bug and fix it.
Unless you plan on having 1 satellite per airplane, something tells me it's harder to constrain the FOV than you might suggest. There's also the small problem of the energy, complexity, & weight of having motorized parts on the satellite (or fine-grained attitude control for the satellite itself to track the craft).
Agreed, my point is it's a lot harder than tiagod made it sound.
It also doesn't account for some kind of mobile jammer making it inside the cone, particularly if it's staring at an adversarial nation where secure comms would be needed the most, but the adversary would have freedom of movement.
While I agree that shared nothing wipes the pants performance-wise of shared state, surely the penalty you've outlined is only for super short lived connections?
For longer lived connections the cache is going to thrash on an inevitable context switch anyway (either do to needing to wait for more I/O or normal preemption). As long as processing of I/O is handled on a given core, I don't know if there is actually such a huge benefit. A single pinned thread for the entire lifecycle has the problem that you get latency bottlenecks under load where two CPU-heavy requests end up contending for the same core vs work stealing making use of available compute.
The ultimate benefit would be if you could arrange each core to be given a dedicated NIC. Then the interrupts for the NIC are arriving on the core that's processing each packet. But otherwise you're already going to have to wake up the NIC on a random core to do a cross-core delivery of the I/O data.
TLDR: It's super complex to get a truly shared nothing approach unless you have a single application and you correctly allocate the work. It's really hard to solve generically optimally for all possible combinations of request and processing patterns.
I would be surprised if they were particularly expensive. There's a GIL, so you don't get concurrency benefits -- but that mainly makes them behave like async.
> Half the kernel is still built by individuals: people using gmail.com, personal domains, or university emails. The "corporate takeover" narrative is overstated. Companies contribute heavily, but the kernel remains a genuinely collaborative project.
Isn't the assumption here flawed? Someone may be employed by a corporation but still use their gmail/personal domain/university domain. This needs to be cross-correlated against some secondary source of employment data to give a more accurate picture.
That's an assumption in the exact opposite direction. GP is assuming that if someone commits while employed by a company then that company paid completely for that commit, while you're assuming that in that case the company "probably" didn't pay for the whole commit.
Either way, the article's conclusion seems to be insufficiently supported.
I actually specifically didn’t state it one way or the other, just highlighting the potential for undercounting. I think there’s some potential for overcounting if you assume employer paid for commits, but less so if you constrain to those employed to work on the Linux kernel - I don’t know if many people would be working “spare time” on the same thing they’re getting paid for.
Maybe reversible computing will help unlock several more orders of magnitude of growth.
reply