Real-time collaboration: Add new REST endpoints, setting, and registered post meta

[chriszarate]

Description

Trac ticket: https://core.trac.wordpress.org/ticket/64622

In Gutenberg, we have added support for real-time collaboration using CRDT documents (via the [https://yjs.dev/ Yjs library]). This work has suggested the following additions to WordPress:

1. A default "sync provider" based on HTTP polling that allows collaborators to share updates with each other. Previously, we relied on WebRTC connections between collaborators for this purpose, but it proved unreliable under many network conditions.

   • Our solution is designed to work on any WordPress installation.
   • HTTP polling is the transport we identified as most likely to work universally.
   • Given the isolation and lifecycle of PHP processes, updates must be stored centrally in order to be shared among peers. We have chosen to store updates in post meta against a special post type, but alternate storage mechanisms are possible.
   • Collaborative editing can involve syncing multiple CRDT documents. To limit the number of connections consumed by this provider, requests are batched.
   • To prevent unbounded linear growth, updates are periodically compacted.
   • To avoid excessive load on lower-resourced hosts, this provider will benefit from usage limits (e.g., a maximum of three connected collaborators) enforced by the client (Gutenberg).

2. A new registered post meta that allows Gutenberg to persist CRDT documents alongside posts.

   • This provides all collaborators with a "shared starting point" for the collaborative session, which avoids duplicate updates.
   • Content stored in the WordPress database always remains the source of truth. If the content differs from the persisted CRDT document, the CRDT document is updated to match the database.

3. A new Writing setting that allows users to opt-in to real-time collaboration.

   • Enabling real-time collaboration disables post lock functionality and connects users to the sync provider.

4. A behavior change to autosaves is needed. When the the original author is editing a draft post (post_status == 'draft' OR 'auto-draft') and they hold the post lock, the autosave targets the actual post instead of an autosave revision. This puts the post data and the persisted CRDT document out of sync and leads to duplicate updates. When real-time collaboration is enabled, all collaborators must autosave in the same way.

This PR provides a proposed implementation of the changes above. This corresponding Gutenberg PR moves the work from the experimental directory to lib/compat:

WordPress/gutenberg#75366

Cumulative work to add this functionality can be found using this label:

https://github.com/WordPress/gutenberg/issues?q=label%3A%22%5BFeature%5D%20Real-time%20Collaboration%22%20is%3Apr

Testing instructions

1. Check out this branch and follow the setup instructions in the README.
2. Settings > Writing > Enable real-time collaboration.
3. Open a post for editing in two browsers / browser tabs.

[github-actions]

Hi @chriszarate! 👋

Thank you for your contribution to WordPress! 💖

It looks like this is your first pull request to wordpress-develop. Here are a few things to be aware of that may help you out!

No one monitors this repository for new pull requests. Pull requests must be attached to a Trac ticket to be considered for inclusion in WordPress Core. To attach a pull request to a Trac ticket, please include the ticket's full URL in your pull request description.

Pull requests are never merged on GitHub. The WordPress codebase continues to be managed through the SVN repository that this GitHub repository mirrors. Please feel free to open pull requests to work on any contribution you are making.

More information about how GitHub pull requests can be used to contribute to WordPress can be found in the Core Handbook.

Please include automated tests. Including tests in your pull request is one way to help your patch be considered faster. To learn about WordPress' test suites, visit the Automated Testing page in the handbook.

If you have not had a chance, please review the Contribute with Code page in the WordPress Core Handbook.

The Developer Hub also documents the various coding standards that are followed:

• PHP Coding Standards
• CSS Coding Standards
• HTML Coding Standards
• JavaScript Coding Standards
• Accessibility Coding Standards
• Inline Documentation Standards

Thank you,
The WordPress Project

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• The Plugin and Theme Directories cannot be accessed within Playground.
• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[pkevan]

Is there any tests we can add for the default provider and new functionality?

[pkevan]

Can we also provide some documentation on how to override the provider given it's known limitation listed above?

[TimothyBJacobs]

Why isn't each room a post?

[TimothyBJacobs]

This seems like it should be a responsibility of the storage layer. If I want to swap with something that can retrieve results ordered by time, we shouldn't have to load all updates always to do that.

[chriszarate]

I've updated the storage interface and class to take this approach in 9e5cf96. It was a little trickier than I thought, because compaction is an external concern and shouldn't be implemented by storage.

[chriszarate]

Why isn't each room a post?

No strong reason. Mostly to avoid creating a large number of posts. Each room corresponds to a synced CRDT document (representing a WordPress entity). Currently supported entity types:

• posts (any post type)
• taxonomy collections

In the future, this could be expanded to include all database entities, including individual taxonomy terms. We might also create rooms to share awareness data at the page or screen level.

We intend to enforce limits on this default provider, but probably applied to simultaneous collaborators and not on the overall number of synced entities. Sharing a single post felt prudent but happy to reconsider! I know wp_postmeta#meta_key is indexed but I don't have a grasp on the performance tradeoff at large scale.

[westonruter]

It could be that the wp_insert_post() fails. In that case, null is returned.

Suggested change

	* @return int Post ID.
	*/
	private function get_storage_post_id(): int {
	* @return int|null Post ID.
	*/
	private function get_storage_post_id(): ?int {

[chriszarate]

Thanks for all these! I believe I addressed them all in 0c5a7fd and 9e5cf96.

However, I decided to loosen the type for the actual sync update to mixed, because this data should be opaque to the storage mechanism (beyond serializability). Let me know if you have thoughts!