> I recall being frustrated with that, and half-jokingly proposing that it would be good to move to a more custom storage solution or have custom indexes next to Git’s storage itself.
There's surprisingly very few options for databases with good structural sharing. I think Pijul's Sanakirja targets this role (_"More importantly, I needed to fork tables efficiently, without copying anything"_), but it's intended for single-node operation.
> So how does not using refs in the dev loop help?
I think it's more of a paradigm shift —
- Producers/consumers of refs must switch from treating ref reads/writes as strongly consistent, and instead move to an eventual consistency model.
- For automation, I would recommend that the "ref resolution" API and the "fetch" API be different: then it becomes obvious that the result of ref resolution may be stale by the time you try to fetch the commit.
- Automation should not directly be able to run git fetch <remote> <ref>. Instead, they should have to call some "git-show-ref" service to resolve the ref into a hash according to the global ref-map, and only then run git fetch origin <hash>.
- It should be understood that the global ref-map may have propagation delay between reads and writes. If you want to be sure to fetch a specific commit, then you had better pass around the exact commit hash.
- (Fetching by hash doesn't even work in Git unless you enable a certain server-side config option IIRC, hence this is a significant mindset change.)
With GitHub, every push writes to one or more refs, and every ref synchronizes on the same global ref-map resource to provide strong consistency. This is a fatal mistake: your push throughput is immediately bounded by how efficiently you can synchronize+mutate that global resource.
In _Git at any scale_, Cursor remarks on this:
> With this design, we ensure that every push is fully synchronized across all the replicas. Reads (fetches, clones) can then be safely routed to any single replica, because every replica is always up to date. [...] We linearize _all_ pushes
They've successfully stored the ref-map separately from the objects. But there's still no reason to linearize _all_ pushes.
If you have a large repo, but most users are not pushing to the same branches, you _still_ technically get contention on the ref-map. It's just very simple and efficient to append entries to a WAL and rely on S3 to handle distributed consensus.
Scion also doesn't adopt the linearized-pushes model:
- If you imagine that every machine/agent has its own checkout, and every commit is made available globally, then the number of "pushes" increases by one or two orders of magnitude, at which point linearizing pushes starts to sound scary again.
- It's actually pretty hard to escape, because the "default" workflow in Git is to push/fetch refs, not objects. If every consumer pushes/fetches refs by default, it adds a lot of unnecessary synchronization to the system as a whole. It's best to strongly encourage consumers to pass around hashes explicitly, and maybe ref names as a non-normative identifier.
- You could still offer refs as a concept, but if you're going to redesign your entire source control system for scalability, why prematurely pessimize?
Humans of course use refs for a few reasons:
- To keep track of "local" or "draft" work; but this can be done without refs, such as with the Mercurial "phase" model.
- To collaborate on named lines of work with others. This genuinely does require some kind of shared identifier, whether it's a ref name, a code review identifier, a change ID, or something else. But humans are usually not stressing the source control infrastructure by doing this.
- I suspect Scion has some kind of notion of "code review" identifier for its commits, similarly to Meta/Phabricator, but, if so, it's not detailed in the article.
I'll note that there are a few other options to a globally synchronized ref-map:
- Locking only _individual_ refs for push.
- This probably isn't feasible in the Git architecture since the source of truth for ref-maps is split across many data stores (loose refs, packed refs, ref-table) and it would be hard to apply a fine-grained lock.
- I doubt this approach is that useful or efficient in practice compared to the linearized WAL approach.
- Make refs into a conflict-free data structure, like in the Monotone branching model.
- If you change your source control model to embed the effective ref _in_ the commit (such as a "change ID" concept), and apply the Monotone branching model with some kind of server-side indexer (that you hopefully already have), then you get conflict-free globally-distributed refs for "free".
- In Jujutsu, there was some discussion about what the semantics should be if you have a draft commit and a public commit with the same change ID but no common descendant. Should it be marked as a conflict? Should the system assume that the public commit is the "canonical" one (i.e. it was "merged upstream")? What if there are multiple public commits with the same change ID?
>> Beyond these, there are standard workarounds for large monorepos, like internal git mirrors and repository snapshots (seeds) kept in blob storage. We’ve used all of them—at our peak, we operated several hundred internal git mirrors to deal with the load.
>
> At Databricks, we do something similar with reference repos which are updated every ~2 hours, and we have fewer than 10 git mirrors to deal with CI load, despite the repo size being within the ~same order of magnitude
At my work, we also have a large monorepo and also ~10 Git mirrors, so it was also surprising to hear "hundreds" from Anthropic. But I believe our CI folks invented their _own_ mirroring system specifically for certain CI jobs which fan out to many machines to run tests in parallel.
- In that case, a leader does the first clone from our Git mirror, and then the 10–100 shards/followers clone from that leader.
- It's possible that they optimize this by creating a single packfile/bundle containing the fetch delta and distributing that to each follower; in that case, they're effectively leveraging "data locality" of the fetch by noting that every follower needs to get the exact same commit.
- It's possible that if the main Git mirror system served this traffic directly that we'd have closer to "hundreds" of mirrors?
> One of the major problems with the blog post is that it is unclear as to what primary purpose it is meant to serve and who the target audience is.
I personally feel that the post is not "novel" research, but the purpose is clear as an engineering report:
- It "reproduces" the results of the original Mononoke "research" and confirms "yes, this approach performs as advertised!".
- Reproducing research is good! Of course it _obviously_ scales because it works at Meta, but sometimes there's "secret sauce" somewhere else in the architecture that's non-trivial, and it's not always surfaced in the original report.
- In this case, you could say that Mononoke depends on magic, infinitely-scalable object storage, but this is no longer remarkable because everyone has access to it these days!
- It presents work in a different form to make it accessible to a different audience.
- Sometimes there is value in rehashing existing research in a way that is easier to grasp, even though no "new" ideas are introduced.
- For better or for worse, the report certainly succeeded in introducing the ideas to a different audience.
I agree that the post would have been substantially enhanced by concrete links to code. Since they're not open-sourcing Scion, then referring to the Mononoke codebase would have been helpful.
- For example, I have source_control.thrift bookmarked (the Mononoke RPC interface) and I refer to it regularly.
13 points
Mentions
my PR got merged into meta's pyrefly github.com/meta/pyrefly/commit/cc2856cc39a199d26c…
> I recall being frustrated with that, and half-jokingly proposing that it would be good to move to a more custom storage solution or have custom indexes next to Git’s storage itself. There's surprisingly very few options for databases with good structural sharing. I think Pijul's Sanakirja targets this role (_"More importantly, I needed to fork tables efficiently, without copying anything"_), but it's intended for single-node operation. > So how does not using refs in the dev loop help? I think it's more of a paradigm shift — - Producers/consumers of refs must switch from treating ref reads/writes as strongly consistent, and instead move to an eventual consistency model. - For automation, I would recommend that the "ref resolution" API and the "fetch" API be different: then it becomes obvious that the result of ref resolution may be stale by the time you try to fetch the commit. - Automation should not directly be able to run git fetch <remote> <ref>. Instead, they should have to call some "git-show-ref" service to resolve the ref into a hash according to the global ref-map, and only then run git fetch origin <hash>. - It should be understood that the global ref-map may have propagation delay between reads and writes. If you want to be sure to fetch a specific commit, then you had better pass around the exact commit hash. - (Fetching by hash doesn't even work in Git unless you enable a certain server-side config option IIRC, hence this is a significant mindset change.) With GitHub, every push writes to one or more refs, and every ref synchronizes on the same global ref-map resource to provide strong consistency. This is a fatal mistake: your push throughput is immediately bounded by how efficiently you can synchronize+mutate that global resource. In _Git at any scale_, Cursor remarks on this: > With this design, we ensure that every push is fully synchronized across all the replicas. Reads (fetches, clones) can then be safely routed to any single replica, because every replica is always up to date. [...] We linearize _all_ pushes They've successfully stored the ref-map separately from the objects. But there's still no reason to linearize _all_ pushes. If you have a large repo, but most users are not pushing to the same branches, you _still_ technically get contention on the ref-map. It's just very simple and efficient to append entries to a WAL and rely on S3 to handle distributed consensus. Scion also doesn't adopt the linearized-pushes model: - If you imagine that every machine/agent has its own checkout, and every commit is made available globally, then the number of "pushes" increases by one or two orders of magnitude, at which point linearizing pushes starts to sound scary again. - It's actually pretty hard to escape, because the "default" workflow in Git is to push/fetch refs, not objects. If every consumer pushes/fetches refs by default, it adds a lot of unnecessary synchronization to the system as a whole. It's best to strongly encourage consumers to pass around hashes explicitly, and maybe ref names as a non-normative identifier. - You could still offer refs as a concept, but if you're going to redesign your entire source control system for scalability, why prematurely pessimize? Humans of course use refs for a few reasons: - To keep track of "local" or "draft" work; but this can be done without refs, such as with the Mercurial "phase" model. - To collaborate on named lines of work with others. This genuinely does require some kind of shared identifier, whether it's a ref name, a code review identifier, a change ID, or something else. But humans are usually not stressing the source control infrastructure by doing this. - I suspect Scion has some kind of notion of "code review" identifier for its commits, similarly to Meta/Phabricator, but, if so, it's not detailed in the article. I'll note that there are a few other options to a globally synchronized ref-map: - Locking only _individual_ refs for push. - This probably isn't feasible in the Git architecture since the source of truth for ref-maps is split across many data stores (loose refs, packed refs, ref-table) and it would be hard to apply a fine-grained lock. - I doubt this approach is that useful or efficient in practice compared to the linearized WAL approach. - Make refs into a conflict-free data structure, like in the Monotone branching model. - If you change your source control model to embed the effective ref _in_ the commit (such as a "change ID" concept), and apply the Monotone branching model with some kind of server-side indexer (that you hopefully already have), then you get conflict-free globally-distributed refs for "free". - In Jujutsu, there was some discussion about what the semantics should be if you have a draft commit and a public commit with the same change ID but no common descendant. Should it be marked as a conflict? Should the system assume that the public commit is the "canonical" one (i.e. it was "merged upstream")? What if there are multiple public commits with the same change ID? >> Beyond these, there are standard workarounds for large monorepos, like internal git mirrors and repository snapshots (seeds) kept in blob storage. We’ve used all of them—at our peak, we operated several hundred internal git mirrors to deal with the load. > > At Databricks, we do something similar with reference repos which are updated every ~2 hours, and we have fewer than 10 git mirrors to deal with CI load, despite the repo size being within the ~same order of magnitude At my work, we also have a large monorepo and also ~10 Git mirrors, so it was also surprising to hear "hundreds" from Anthropic. But I believe our CI folks invented their _own_ mirroring system specifically for certain CI jobs which fan out to many machines to run tests in parallel. - In that case, a leader does the first clone from our Git mirror, and then the 10–100 shards/followers clone from that leader. - It's possible that they optimize this by creating a single packfile/bundle containing the fetch delta and distributing that to each follower; in that case, they're effectively leveraging "data locality" of the fetch by noting that every follower needs to get the exact same commit. - It's possible that if the main Git mirror system served this traffic directly that we'd have closer to "hundreds" of mirrors? > One of the major problems with the blog post is that it is unclear as to what primary purpose it is meant to serve and who the target audience is. I personally feel that the post is not "novel" research, but the purpose is clear as an engineering report: - It "reproduces" the results of the original Mononoke "research" and confirms "yes, this approach performs as advertised!". - Reproducing research is good! Of course it _obviously_ scales because it works at Meta, but sometimes there's "secret sauce" somewhere else in the architecture that's non-trivial, and it's not always surfaced in the original report. - In this case, you could say that Mononoke depends on magic, infinitely-scalable object storage, but this is no longer remarkable because everyone has access to it these days! - It presents work in a different form to make it accessible to a different audience. - Sometimes there is value in rehashing existing research in a way that is easier to grasp, even though no "new" ideas are introduced. - For better or for worse, the report certainly succeeded in introducing the ideas to a different audience. I agree that the post would have been substantially enhanced by concrete links to code. Since they're not open-sourcing Scion, then referring to the Mononoke codebase would have been helpful. - For example, I have source_control.thrift bookmarked (the Mononoke RPC interface) and I refer to it regularly.
@serialx_net @mitsuhiko Why in the world would someone name this pyre, out of all possible names? meta/pyre-check
Welcome to the new game format ZSTD—so long, lizard, and hello ZebraS-Tailed lizarD meta/zstd With this new format, for example the game Control Resonant finally works on firmware versions 12.xx and 13.xx 🦎 +🦓 = 🐉 t.co
让 agent 搭界面，拼出来的组件风格各异？ astryx 是 Meta 开源的设计系统：150+ 可访问组件、暗色模式，还有 CLI，人和 agent 用同一套搭界面。 让 agent 出界面的团队来说，同一套组件库比各写样式稳。 先用它搭什么页面，后台还是落地页？ #设计系统 #开源 @grok meta/astryx