Very, very cool exercise. For anyone who hasn't been following APFS much though, worth knowing that Apple has stated in their AFPS FAQ [1] that they do intend to directly "document and publish the APFS volume format specification" themselves later this year. Presumably (as they do with other stuff they develop) they want to be able to more easily make breaking changes during the shakedown period of initial public deployment (first on iOS right now, later on Macs, though maybe not a default option until 10.13 this Summer/Fall). That'd be the most helpful in getting read/write into FUSE and other platforms should anyone choose to do so.
They said the same about Facetime and iMessages but never opened or documented those, so let's see when and if it actually happens. Apple has a history of false promises when it comes to opening and documenting their protocols & data formats.
Edit: I misrecalled that iMessage was announced as something they were opening.
I guess you missed the whole lawsuit thing [0] that forced them to make (initially P2P) Facetime completely centralized at which point it was pointless to open it up.
PS: If you're gonna hate on Apple, at least find some valid points about them to hate instead of just spreading bullshit.
Apple did say they were opening it up; then they didn't. How's that bullshit? Even if they did have a legal excuse, that doesn't make it any less true. It could happen again.
I think it's a bit unfair to paint a genuine and honest intention, that is even pursued as far as a court of law, but which is unexpectedly prevented by a third party a 'false promise'.
They never promised to open up iMessage. The story behind FaceTime is a little bit more complicated. Steve Jobs said he would open it up without asking the engineers and lawyers who were just as surprised as anyone else about his announcement.
Apple's relationship with open source is pretty average. In the end, they use open source strategically. If there's no benefit to open-sourcing something, they won't.
Clang, WebKit and Swift are "proper" open source projects, done in the open. But there aren't many of those. They have no open source apps, and most frameworks such as Cocoa, AppKit etc. are completely proprietary.
Some of their open source efforts have been half-hearted; Darwin was enthusiastically embraced by the community, then essentially shut down by Apple. The XNU sources are APL, but there's nothing open about its development. I/O Kit is closed, I believe.
A lot of companies routinely open-source lots of little things that could easily have stayed internal, but are genuinely useful to everyone else. (Admittedly, I think about 30% of them are Go logging libraries, but the point still stands.) I.e., open source by default as opposed to being the exception.
That, of course, can be a strategy. A company can look cooler and attract talent by developing everything in the open.
Webkit was open-source long before anticipating killing Flash. I think, like Darwin, it was originally open-source because it had to be, due to its origin in an existing open-source project that required derivatives also be open-source.
They don't but they both perform much better than Apple on the subject.
React is one of the leaders in JS dev and several other Facebook projects are quite useful.
Microsoft is the company with the biggest number of open source contributors and several of their projects like TypeScript are extensively used and they are the biggest contributors to LINUX https://octoverse.github.com/
Apple made Swift an open source language that runs only on their proprietary devices and that's it...
Swift does not run "only on their proprietary devices". It runs on at least Linux as well.
I'm not saying that Apple is better, or denying that Facebook does a 'better job', but instead just reinforcing the concept that all these technology companies don't open source out of the goodness of their heart, but instead for the same strategic decisions that Apple does it for.
They do have at least one open source application: https://www.cups.org/
it's fully open-source, and wholly owned by Apple, however they bought it, they didn't invent it.
They also lead development of LLVM and clang, which is a _huge_ project, and very useful (although sometimes without full realization of that fact) to the Open Source ecosystem.
And those lawsuits forced Apple to re-engineer how Facetime worked, from peer-to-peer direct connections to a more Apple-intermediated model[1]. At that point, creating an open standard is less interesting, since you can only communicate with folks using the same "relay server" network.
Regarding Apple’s Open Source strategy, the WebKit origin story is a major disaster.
They forked an existing project, only did sourcedumps every few months, often stripped of all comments, refused to work with the original devs that they had forked from (the KDE KHTML engine), etc.
Open sourcing Swift, though, probably benefits Apple more than it does the community. Sans the Apple UI, it doesn't create some opportunity to lose a sale or have Apple like consumer facing apps show up on some other platform. Apple apps already mostly rely on non-apple backend servers.
Opening up iMessage or FaceTime would have been more of a statement. Either would allow interoperability on platforms they don't control.
Is there something significant they've open sourced, despite some perceived risk?
> Open sourcing Swift, though, probably benefits Apple more than it does the community.
I am no fan of Apple myself, but it's worth giving them credit where credit is due, they did everything right when open-sourcing Swift and are very responsive to the community, there's really no downside for anybody here, so why the need to criticise? It only needlessly provides the wrong incentives.
It is not criticism. It's an observation, followed by a question. I'm honestly interested in why Apple has the reputation it does around open source. The replies in the sub thread have been enlightening.
Apple gets this bad rap for OSS, because of Google's good PR a few years back on how "Android was open." But it's not like Google has a better track record of supporting OSS when it comes to products that influence their bottom line. Adwords, Google Search, AdSense, Analytics, etc are not open source, and no one ever complains about that.
The comment you're replying to, and all the parents up to the root, like most comments here made no mention of Google. It was a discussion of Apple on its merits.
Why do we seem to tie our personal identities to corporate affiliations, to the point of bringing up corporate rivalries out of context and taking sides in them?
Those are good examples. It was an honest question though, not a proclamation that Apple hasn't open sourced anything.
Edit: Maybe Google is getting more credit because of uptake? Chromium, golang, protobufs, material design, etc. So perhaps not how much was open source, but how much of it ended up in non Google projects?
Chromium was a fairly recent fork of WebKit. Webkit was already being adopted by every browser vendor except MS and Mozilla.
As far as Golang, I doubt that it has the uptake ofClang (even used by MS), CUPS, or Bonjour.
Yes I realized that CUPS was open source before being acquired by Apple but the copyright was owned by Sweets and they could have closed all future enhancement over the last decade.
You're talking about Blink. Chromium is not a rendering engine, but a full browser. Chrome is just chromium with branded assets and a few proprietary features that link it to Google Services.
It was first released in 2008, only 3 years after WebKit was open sourced, and about the same it was starting to gain traction outside Apple. Most of the browsers which didn't have their own engine back then were actually using Gecko or embedding an IE webview. Google embracing WebKit was a huge factor in pushing it to become the mainstream rendering engine.
They didn't open WebKit. It's a fork of the LGPL'd KHTML code base. Apple is legally obliged to keep it open, so whatever anyone thinks about Apple's open source record using WebKit as an example makes little sense.
Eh, not quite. Apple could satisfy its LGPL obligations by throwing source dumps of WebCore over the wall - without releasing the rest of WebKit, exposing SVN history, or having a public bug tracker. In fact, for the first two years of Safari's life, that's exactly what they did. Then in 2005 they opened it up properly, voluntarily:
Of course, they only cared about PR: by pure coincidence, they "voluntarily" changed development model on webkit right after several news outlets had picked up the umpteen blog post by a grumbling KDE developer.
Yes, Apple historically doesn't give a shit about the OSS community unless they can exploit it or it becomes a dangerous nuisance. And this is why it gets a bad rep in OSS circles.
Google has nothing to do with this. Apple get their bad rep from their behavior on Darwin, webkit and the GNU bait-and-switch (first adopted with great fanfare, then left to rot and eventually purged mercilessly). The fact that they strenuously refused to let some of their desktop goodness leak back into the OSS world, from the very first day OSX was announced, is also an old pain point.
The context bubbles up to the post that said "Apple has a history of false promises when it comes to opening and documenting their protocols & data formats". That comment was down voted, but had me curious.
I was asking for examples of something they open sourced that would have exposed something like that, something more risky to the walled garden concept. I perhaps could have worded it better, but HN does seem to react to any questioning of Apple with a defensive posture.
something more risky to the walled garden concept. I perhaps could have worded it better, but HN does seem to react to any questioning of Apple with a defensive posture.
What has any company open sourced that would cause them to loose their competitive advantage?
Certainly, Android is more open than Apple's mobile OS. Google does hold some important bits back. But, still, it is "more risky". Hence the question. I may be missing other areas where Apple was open sourcing things that were more core to them.
Android is "more open" except for the parts that Google cares about - Google Play Services, and all of the other open source components that Google is abandoning.
I get the point that you don't like this line of inquiry, but "something more risky" isn't "definitely lose your competitive advantage". And, again, it's a question, not an accusation. I'm curious...trying to understand the Apple reputation in the open source area, how much of it was deserved or not.
Certainly, Android is more open than Apple's mobile OS. Google does hold some important bits back. But, still, it is "more risky". Hence the question. I may be missing other areas where Apple was open sourcing things that were more core to them.
It's not Apple's fault that they didn't open up FaceTime. They absolutely were going to, and then they were sued by a patent troll (VirtnetX), and was forced to change how FaceTime works so it relays through Apple's servers instead of going peer-to-peer. Since it was now going through Apple's servers, there was no point in opening up the protocol.
IRC also needs someone to run a server. Just because a protocol isn't p2p there's plenty of reason to open it. You can independently develop third party clients, use your own server instead of Apple's etc.
They simply lied. They never said "We'll open Facetime, but only if it's implemented as a p2p protocol & client"
Changing one's mind based on changing circumstances isn't lying- it's common sense. The 'thing' that Steve Jobs wanted to publish as an open protocol stopped existing due to a lawsuit, the 'thing' that replaces it shared only its name.
I think that's an absurdly low standard to apply to corporate speech.
The CEO of a multi-billion dollar company gets on stage during their main annual product conference and says a newly announced product is going to become an open industry standard.
This turns out not to possible in exactly the way the company had planned due to a lawsuit, so the company drops the plan entirely.
That's a lie by definition. You asserted that you'd do something and then you didn't do it. Does breaking a promise imply that there can't be any mitigating circumstances? No, I'm sure there were some reasons for why FaceTime wasn't opened.
People & companies make product buying decisions based on things like these, e.g. some company might have bought a bunch of iPads for video conferencing expecting that in a couple of years they'd be inter-operable with any device they could buy, including non-Apple devices.
This excuse that they couldn't open the p2p thing they initially planned to open is absurd. Apple could absolutely have just opened & standardized what FaceTime ended up being, i.e. an open video chat client/server allowing anyone to run a server. I.e. some sort of IRC or Jabber for video messaging.
Even if they couldn't open or release some specific component due to patent concerns they could still have opened & standardized almost everything else, see e.g. how open source video players ship pluggable codec support without direct support for some proprietary & patented codecs.
Bringing this back to my original comment on APFS, do I think they won't open that? No, they probably will. I just thought I'd point out that Apple has a documented history of abandoning their open source / standardization plans after making some very public announcements about them, so people should be wary of counting their chickens before they're hatched when it comes to Apple.
> Apple could absolutely have just opened & standardized what FaceTime ended up being, i.e. an open video chat client/server allowing anyone to run a server. I.e. some sort of IRC or Jabber for video messaging.
But why? There's literally no point to doing that. The whole point of opening up FaceTime was so people using non-Apple devices could talk to people using Apple devices. After the lawsuit that became no longer possible.
Telling a lie is to say something you know isn't true. If they honestly intended to open source FaceTime and then changed their mind (for whatever reason) that is not a lie by any definition.
But I agree with you that it casts doubt on their determination to follow through on their open source promises. And that sort of doubt is extremely detrimental to any uptake of open sourced software.
The whole point of opening it up is so other people's clients can talk to FaceTime users and vice versa. But Apple's not going to provide their own servers for third parties. And without that, there's no interoperability, which means there's no point in opening up the protocol.
Yet they said the same about Swift, ResearchKit, etc. Those are successful open source projects. FaceTime is the only recent project they said they would open up and didn't...and that arguably wasn't their fault.
For those who, like me, don't follow developments in this area closely:
APFS is Apple's new filesystem, intended to be used across all of their devices (Macs, iOS, watchOS, etc). For Macs, it will be replacing HFS+, a direct descendant of the Hierarchical Filing System which has been used since 1985. (The original Macintosh Filing System, from 1984, was suitable only for floppy disks; among other things, it was a flat filing system and didn't really support folders.) The main improvements HFS+ made over HFS were support for larger files and Unicode; Apple has since added some more features but the core filesystem remains a 30-year-old format which was designed for early hard disks.
Great link! I'm very interested in the mention of per file keys, one for data and one for metadata. I wonder what data structure they store these in. It sounds a lot like cryptree [0].
Kaitai which was used as a parser for this looks pretty interesting. I need to reverse engineer some of the Allen Bradley Ethernet Industrial Protocol and its client communication. This seems pretty useful for documenting that.
Holy shit this is so cool. I will never have a need for this but it's really amazing to poke around with. It kind of feels like Wireshark for files instead of packets.
They also have a web version[0] with a bunch of example files if you want to check it out without installing it locally.
Apple should publish something like TN1150 for APFS. several years ago, I had to implement HFS+ for an embedded system, and TN1150 proved very helpful, saved me a lot of hassle.
2^64 nanoseconds is > 580 years. Even cutting it in half for signed values is still likely 10X the expected lifetime of this particular iteration of APFS.
The granularity is probably useful for establishing an ordering (even somewhat arbitrarily) for highly concurrent file system operations, and thus potentially skipping a bunch of expensive synchronization.
Found this: https://github.com/jbenet/nanotime. No idea if related in any way or used for APFS but it was interesting enough for a quick read. This one is MIT licensed. A little more searching I found a variant but it's done with big-endian so it's much different.
Awesome research, would love to be able to do that.
Some things that came to mind: I like that some of the block types are "acronyms" of their meanings: 5 = S(pace man), B = B(-tree), C = C(heckpoint). Also, I wonder why the volume superblock magic is "APSB", while the root (container) magic is "NXSB"; maybe this refers to NextStep? Seems a bit anachronistic, but I don't know.
No, nothing like that necessary. Data remains in the same spot, and during the conversion new metadata and file tables/pointers are created in APFS format. Then the volume is remounted using the new format, and the old hfs stuff goes bye-bye.
Does anyone know or can speculate on the purpose of the 4 "alignment" bytes in the node block header? Are they for alignment (in which case it's strange to have them at the beginning instead of the end)? Or are they describing alignment (in which case it's strange to need 32 bits)?
Common in the mainframe world, like VSAM. That would be variable length record oriented files, versus stream files. Someone did do this for GEOS: http://justsolve.archiveteam.org/wiki/GEOS_VLIR , but I've never seen that for anything remotely unix like.
I wasn't even after a conclusion or anything. But since there isn't a footer I just thought maybe my internet connection died and it only loaded a portion of the page.
There are a couple features missing from HFS+. Some of these are implemented in APFS:
* Case sensitivity
* Clones extents / ranges (reflinking)
* Holes in the middle of files
* ACLs
* Collapsing file ranges
* Per file compression and per file encryption
These are useful for apps like SQLite or other stores which implement a multi user, log structured store.
A quick note to any readers... I'd suggest avoiding a case-sensitive HFS+ boot volume with macOS. I used one for over a year, and it just brings trouble.
The biggest problems I encountered were (1) occasional bugs with Homebrew formulas, and (2) Adobe's applications cannot be installed on a case-sensitive file system.
HFS and HFS+ were never case-sensitive by default on Mac OS. Case-sensitive HFS+ was made available as an option in Mac OS 10.0, but was always glitchy (especially for boot volumes), because many applications depended on case insensitivity.
I think possibly the point of confusion is that very early versions of MacOS X (the original MacOS X server, and some of the MacOS X DPs, anyway) used UFS by default, not HFS+. UFS is case sensitive.
I believe one reason APFS is touted is because the HFS+ data structures have one big global lock, whereas APFS allows for multithreading which ought to improve performance on multi-core machines.
I heard there will be copy-on-write for files. For web devs this would save a lot of space and time when copying files from yarn cache to node_modules.
I wonder what it means for appending data to a large log file. Does the entire log file need to get copied on every write (or just a block of the file?)
I agree to a point, one factor that can help mitigate this however is using a quality SSD where it's onboard memory has ECC, I suspect Apple sees the near future of end user computing as gaining ECC for all memory by standard which is something I've wanting for years but games have been played by CPU manufacturers to prevent the use of high end desktop CPUs in server environments by I believe purposely kneecapping them from using ECC memory (citation probably needed here).
I haven't done any serious analysis of Time Machine (or of APFS for that matter), but I think in general Time Machine also only keeps one copy of files. So APFS isn't worse than the existing situation. Arbitrarily keeping multiple copies blows up the disk requirements. You can intentionally keep old copies around in APFS via snapshots, which is what APFS will probably do when implementing Time Machine.
What Apple added for the existing Time Machine on HFS+ was an extension of the Unix concept of hard links to files (where there is only one copy). They added hard links to directories. Generic Unix avoids doing that for a number of reasons.
Isn't is already true that there is one copy to corrupt? The whole idea of using hard-linked trees (a la rsnapshot [1]) presumes that when a file doesn't change, there is only one copy.
Look, we get it, you bought an Android and you want to use iMessage and FaceTime.
That would be stupid, Apple doesn't make money from users data, Apple doesn't listen to conversations like Google does to sell ads.
So why would they support servers (and there are a lot more of android users than iOS users) for people that didn't bought anything from them? For people that bought Androids to like them? They don't need that.
It has nothing to do with open source, the open source community doesn't need this and Apple couldn't care less about the GNU types after GPLv3 incompatibilizes with their business (while Android OEMs break the license every single day, but the community turns a blind eye, because it's Google)
Also, they would lose a few of their product sales and lots of 30% shares on sticker packs and other Apps for iMessage they sell.
That's not at all why I posted this or where I was going. I do happen to have an Android, but don't use apps other than a web browser and whatever the SMS app is. I have no desire to video chat, or any need for SMS beyond what a Nokia can do.
I am just interested in knowing more about why Apple has the reputation it does around open source. The thread has delivered on that. Also delivered on the idea that apple fans are oddly defensive. Honestly not trolling, but the reaction seems as if I did succeed in that somehow.
---
1. https://developer.apple.com/library/content/documentation/Fi...