Browse Source

Initial documentation

Max Dor 4 years ago
  1. 1
  2. 23
  3. 206
  4. 181
  5. 177


@ -0,0 +1 @@


@ -0,0 +1,23 @@
# The Grid
The Grid is a fork of the Matrix protocol with a different set of core values and a true community management.
It aims to be care more for privacy and much more friendly to self-hosted infrastructures.
## Status
We are currently putting it all together before official kickoff.
See the [Sunrise]( milestone for the current progress.
## Documents
The following bootstrap documents explain what The Grid is in more details, how it came to be, differences with Matrix,
how the protocol will be managed by the non-profit and how documentation should take place.
- [Project Overview](docs/
- [Governing Body](docs/
- [Documentation Principles](docs/
## Contact
Visit us on:
- Matrix
- [](
- [](
- Telegram: [@thegrid_matrix](


@ -0,0 +1,206 @@
# Documentation Principles
## Scope
This document aims to:
- Define the value and importance of documentation for the protocol and in the ecosystem
- Define key terms
- Describe the principles of Documentation processes
- Describe the various types of Documentation
## Concepts
We strongly believe in documentation, not because of the documentation itself, but because of what it represents, the
values it upholds and a possibly near universal recognition of worth for the knowledge it contains.
The rest of this document will describe those in details.
### Proof of Work
The expression ***Proof of Work***, shortened to ***PoW***, will be used throughout this document to represent the effort
made by someone to articulate and communicate meaningfully about relevant matters towards others. It comes from the
[Proof-of-work system](
There is no restriction on the amount of work itself, which can range from near-to-nothing to extensive amounts of time
and money. It will directly depend on the situation and the goal of the debate.
Anyone in the community (including from the boards) who wants to participate in a debate MUST produce ***PoW***.
***PoW*** does not mean minimal amount of knowledge. Rather, ***PoW*** means minimal amount of effort.
We expect back-and-forth argumentation backed by ***PoW*** as people participate in an exchange of some sort, until no
more argumentation can be brought up, which will settle that particular exchange.
### Documentation
The word ***Documentation*** will be used as a generic concept throughout this document.
Any document that seeks to be recognized as *Documentation* must incorporate at least one ***PoW***.
It is defined as something that:
- Is available publicly
- Is directly reachable using a [URL](
- Is a standalone piece of work, or directly incorporated in one at the same location
- Is written in English if text-based
- Incorporates at least one ***PoW*** from the concepts below
- Can be re-used by others to build on
- Provides a publicly accessible channel for feedback
Examples of Documentation:
- A standalone HTML page
- A PDF document
- An [ODT]( document
- A Gogs/Gitea/Github/Gitlab issue
- A comment on an issue
- A Cryptpad document
- A Google Docs document
- A schematic wireframe
- A fully rendered and styled visual mockup of a GUI
Any produced *Documentation* is guaranteed to be reviewed by the governing body in a timely fashion. *Documentation* will
therefore become the ideological currency of the project and ensure everyone who worked to produce one will be able to
"trade" it for something else (e.g. trade it for the review of the documentation).
We make no generic requirements on the format, length or overall wording of *Documentation*, as long as the vast majority
of the community is happy about it. This is on purpose, following on the issues met so far using a too strict proposal
process and we’ll rely on a naturally occurring balance with gentle nudges in the right direction.
The Technical Board will produce a list of officially recommended formats, way of sharing and provide sample templates
for the various fitting use case so people can easily get started. Those will be kept up to date and relevant.
### Implementation
The term ***Implementation*** will refer to the deliverable or the actual product derived from a given *Documentation*.
It is a generic term, it's meaning will change depending on the role/occupation of the deliverer.
Examples of Implementation:
- As a software developer: source code
- As a graphic designer: Wireframes and mockups
- As a User Experience engineer: Guidelines/recommendations
- As a system administrator: Install guides, deployment scripts, etc.
## High-level concepts
To ensure successful *Documentation*, it is essential to respect some core concepts that reflect the value of the protocol,
its openness and the recognition of ***PoW***.
The following concepts are how something will be judged as qualifying as a ***PoW***.
### Extract, organize and materialize thoughts
Our experience shows that the human mind tends to underestimate risks, difficulty level and overall workload.
Therefore we believe that the act of writing the *Documentation* - regardless how small - simply forces one to think further
and in more details about what one tries to accomplish and filters the most basic and unconscious biases.
### State functional problem to be solved
While chat or voice exchanges are good for quick exchanges, it’s easy to lose sight of the actual problem one is trying
to solve and the involved scope, or simply not really know what problem is currently be solved. By stating up front the
problem, the scope is set and can be enforced in further exchanges.
### Facts backing
**This is *Proof of Work***
As this is a technical exercise, it is important that any statement that influences choices, like "it is very likely that
X happens", be backed up by objective, quantifiable and/or qualifiable facts from an independent source.
### Usage feedback and usability
**This is *Proof of Work***
Using or testing software/*Documentation* or any kind of item that comes from a ***PoW*** becomes itself a ***PoW*** once
that experience is documented.
This is typically:
- Software tests in development cycle
- Daily usage from end-users
- Bug reports of software of the protocol specification itself
- Feature request from end-users
In addition to load testing, pen-testing, etc, usability focus groups (whether remote or in person), feedback via surveys
and suggestion forms provide an excellent opportunity to gain new and sometimes unexpected insights. If the same
feature request or complaint appears frequently, it should be considered actionable. Putting together such a compiled
list or results of testing provides actionable next steps.
Ad hoc observations such as “my group found this feature difficult to use” can be logged as an issue and may also be
considered; however compiling quantitative evidence (X% of users could not complete an assigned task) is strongly encouraged.
UX professionals and designers should also be encouraged to submit wireframes, mockups, and process documentation
for future functionality. (Not just "what’s broken?" but "what can be dreamed and built?").
In this instance, a well-developed use case is considered ***PoW***.
### Sources linking
**This is a *Proof of Work***.
As facts must come from external sources, it is critical to include links to those sources, so anyone evaluating the
document can have access to the same information as the author without having to research the data themselves.
### Proposed solution
**This is a *Proof of Work***.
Once the problem to be solved and the scope are documented, one will describe the proposed solution that solves the
stated problem/issue.
Next to the description of the solution, one MUST reiterate in a short conclusion how the solution effectively solves
the problem.
### Rationalisation
**This is a *Proof of Work***.
As one goes through the proposed solution, one will use the various facts and/or linked sources to justify the proposed
solution. One CANNOT rely on common knowledge, as this is a cognitive bias. On the other hand, one is free to rely on
required knowledge.
Example: If describing an HTTP endpoint, one may have to justify the validity of the HTTP choices made (POST vs PUT, etc.),
but one will not have to justify TCP principles.
### Implementation
**This is a *Proof of Work***.
This validates the proposed solution with an actual Implementation and ensures that the theoretical outcome is feasible
in practice, as a real product for real users.
### Peer convincing
**This is a *Proof of Work***.
Proposals should be written in a way that one attempts to convince others that the proposed solution is the correct one,
instead leaving the fact checking work to their reviewing peers.
One MUST NOT assume they are correct to start with. They MUST convince others of the validity of their claims using the
elements described above.
It MUST be easier to read the proposal than to write it.
## Types
*Documentation* will be divided into several types, each having their own requirements and goals.
Except for The Protocol Specification, any of those *Documentation* can be put forward by the governing body or by the
Types are not mutually exclusive and a single *Documentation* can be of several types.
The various types will outline the various stages/steps of the overall process which is a recurring cycle:
1. Guides/Recommendations/Templates and The Protocol Specification allow Implementations to be made
2. Implementations trigger Issues/Bug reports/User feedback
3. Issues/Bug reports/User feedback trigger Research/Analysis
4. Research/Analysis inspire Guides/Recommendations/Templates and Proposals
5. Proposals aim to enhance Guides/Recommendations/Templates and The Protocol Specification - Back to first step.
### Issue/Bug report/User feedback
This kind of *Documentation* will mostly rely on [Usage feedback and usability](#usage-feedback-and-usability).
**Example**: Request to clarify recommended setup for synapse (Github issue).
### Research/Analysis
This kind of *Documentation* will mostly rely on [Facts backing](#facts-backing), [Sources linking](#sources-linking) and
[Peer convincing](#peer-convincing). It can be produced by anyone and will be the primary backing work for other *Documentation*.
**Example**: Matrix Server ACL addition analysis ([Github gist](
### Guide/Recommendation/Template
This kind of *Documentation* will mostly rely on [Usage feedback and usability](#usage-feedback-and-usability) and
[Proposed solution](#proposed-solution).
**Example**: Matrix stack installation guide ([Github repo](
### Proposal
This kind of *Documentation* will mostly rely on [Proposed solution](#proposed-solution), [Rationalisation](#rationalisation)
and [Peer convincing](#peer-convincing).
**Example**: Matrix client auto-discovery ([Google Docs](
### Implementation
This kind of *Documentation* will mostly rely on [Facts backing](#facts-backing) and [Implementation](#implementation).
**Example**: any software implementation following a protocol specification.
### The Protocol Specification
The Protocol Specification will be one of the final products and the most important deliverable for the Governing Body.
As a special authoritative document, its main Proof of Work will be "delayed" and mostly contained in its ability of
[Peer convincing](#peer-convincing) and being backed by two independent [Implementation](#implementation), adding to a
total of three independent *Documentations* (one is the specification, two are the implementations).


@ -0,0 +1,181 @@
# Governing Body
## Scope
This document will outline the high level concepts and guidance principles of the governing body of The Grid that will
be used for the articles of constitution.
This document will NOT go into details of how day-to-day management will happen, which will be left to the discretion
of the Governing Body.
## Legal entity
The Governing Body will be a non-profit organisation, funded in Luxembourg, as an Association sans but lucratif by Max
Dor (author of this document), Daily Manager of Kamax Sarl, and two other people (TBC, people that are known personally,
but not necessarily technical), as legally required.
Non-profit status will ensure that all activities and incomes must be legally targeted to fulfil the missions of the
non-profit without being subdued by making a profit for its shareholders, as opposed to a for-profit which its sole
purpose is to make profit and distribute it to its shareholders.
Non-profit gives a better framework for donations or voluntary work, without the need for extensive accounting or legal
requirements overall, giving much more freedom in the day-to-day running of the organisation and helps keeping management
costs to a minimum, which would otherwise add up to several thousands Euro per year.
The legal entity will be used to centralize ownership of various protocol IP materials, like Internet domains, trademarks,
logos. The source code of any tools built by or on request of the legal entity will remain the IP of their respective owners.
## Missions
The Governing Body will have the following missions:
### Regular protocol specification releases
A protocol is only defined by its specification, and its delivery in a clear, comprehensive, validated and usable format
by the widest technical audience is the most important mission.
The Governing Body exists in fine for the sole purpose of this deliverable and should be judged on it.
### Provide adequate environment, tools and hosting
The Governing Body will provide:
- A documented vision for the protocol
- What is reasonably necessary for its boards to fulfill their responsibilities
- What is reasonably expected from the community about such entity with a similar mission
- End-users software, tools and services to showcase the protocol usage and goals
- Development software, tools, documentation and services to ensure independent implementations are possible and straightforward
- Means and services to discuss, exchange, and learn about the protocol
- Guidelines and processes for protocol changes and community contributions, ensuring these are always welcoming,
as easy as possible, and as rewarding as possible.
### Community management
- A Community Manager which will always be available and reachable
- The community is proactively kept up to date of changes and achievements
- Officially-managed places are properly and timely moderated
- Feedback from The Community, however bad or good, is always taken into consideration
### Protocol fostering
Proactively engage into activities to:
- Make the protocol known by as many people as possible (website, social medias, conferences, etc.)
- Convince end-users to use the protocol (demos, webcast, etc.)
- Convince developers to build on the protocol
- Grow the community at large
## Funding
The Governing Body should only be founded on donations from the community. This will be a critical dynamic for this
project, as it will ensure the community has a clear and straightforward way to show its support and general agreement
with how well the Governing Body is doing.
All donations will be listed and made public. Donors that have made a donation with significant amount (> X€ over X months)
must be publicly disclosed as they can use it as leverage to achieve their own goals. Donations MUST NOT be conditional
to any delivery, service or related item. Donations MUST be without any string attached.
While sponsoring is occasionally acceptable, full disclosure about the transaction, including the contract, MUST be
publicly disclosed. Contracts that conflict with this full public disclosure clause MUST be refused. Signatures, emails,
contact numbers will be removed before public disclosure.
Sales of goods and services are acceptable, as long as:
- They are legally acceptable for the non-profit
- The majority of income is not tied to a minority of customers
- Do not interfere or take priority in any way with the missions and responsibility of the Governing Body outlined in this document
The Governing Body will give full disclosure about incomes on a regular basis (TBD, quarterly?), so it can be audited by the community at large.
## Financing
The Governing Body will be allowed to use its funds to:
- Pay a regular salary to management board members
- Give financial compensation to technical board members, after approval from management board and technical lead, to
ensure they can afford dedicated time and give an incentive to remain a member
- Pay for the various fundamental technical services (Domains, web hosting, etc.)
- Finance trips and related expense to social events which are relevant for the boards’ missions (e.g. FOSDEM stand)
- Finance individual missions with independent contractors/orgs that may have dedicated knowledge into one area, to further
progress the protocol (e.g. a security/crypto consultant for end-to-end encryption, or security audit)
The Governing Body WILL NOT be allowed to provide funds to any entity listed as an employer or of significant involvement
with any of the boards’ members. This will ensure no board member will have an incentive to redirect the Governing Body
money for its own profit.
The Governing Body will give full disclosure about spending on a regular basis (TBD, quarterly?), so it can be audited
by the community at large.
## Leadership
The leadership of the non-profit will be divided into two boards:
- Management board
- Technical board
Membership of those boards are not mutually exclusive, and one person can be a member of both. Each board will have
specific, non-overlapping responsibilities.
We recognize and even value that no one can be 100% neutral and therefore might try to push their own agenda, even if
unconsciously. We see this as a driving force as members would have a reason and a motivation to be there and is not a
negative thing on its own.
As a user, developer, sysadmin, member or donator, it is very important to be able to make an informed decision about
whether one should continue contributing in its own way to the project. This starts by knowing who the people in charge
are, what allegiance they might have and the laws they are bound to.
Therefore, every member of each board must disclose the following on public record:
* Name (not necessarily full legal name, but not an obscure nickname either, like “l33t1337”)
* Employment status with position and employer if applicable
* Significant (paid and/or with decision making level) involvements in other organisations of any kind
* Country of legal residence
Members will be listed on the main website of the protocol project page, along with the latest up to date information.
### Management board
This board will be made from members of the legal entity and will take on the following responsibilities:
- Management of the non-profit legal entity
- Accounting
- Finances
- Promotion of The Grid
- Ensure the governing body’s missions are fulfilled
- Functional (high-level, non-technical) steering of the protocol
- Community management
- Legal member of the non-profit
Membership starts:
- By vote of all current board members
Membership ends if either of the following occurs:
- Of their own will, at any time
- By vote of all other board members
### Technical Board
This board will be made of people who have written implementations of the protocol and thus can have an informed,
knowledgeable and authoritative view about the technical choices to be made and decisions to be taken.
This board’s members will be made of:
- One lead member, called “The Lead”
- An number of other members, called “The Members”
The board will decide on a tie-breaking mechanism for decisions.
The lead will take on a “back-seat” role, only giving his final opinion/choice/vote after everyone else has. The Lead
will have a special role as a spokesperson of the technical board, both towards the management board and the community.
The lead can only be voted in by the management board which MUST seek a written list candidates from the technical board.
The lead can only be voted out by the technical board and MUST, prior to voting, seek a written opinion about it from the management board.
This will ensure that both boards agree on The Lead being a good fit for the role and that potential abuse is kept under control.
The board CANNOT contain more than one member that have financial ties with a specific entity. This will ensure the board
cannot be taken over in the long term by a single employer/corporation/organisation.
- Production of the protocol Specification
- Production and maintenance of an implementation validation tool to certify those implementations as protocol compliant
- Technical steering of the protocol, to fulfil the functional goals
- Review of community proposals
- Proactively ensure enough implementations exist to keep the protocol validated
- Be the main author of at least one maintained (compliant with the current or previous version of the protocol) component implementation
- Submitted at least one protocol change that has been accepted in the official documentation in the last X months
- Commitment to be available X business days per week, except when on justified leave
Membership starts:
- By vote of all current technical board members
Membership ends if either of the following occurs:
- Of their own will, at any time
- By vote of all current technical board members
- By failure to disclose any potential conflict of interest, enforced by the management board if necessary
- By failure to be available as required, enforced by the management board if necessary
- By failure to keep their implementation maintained, enforced by the management board if necessary


@ -0,0 +1,177 @@
# Overview
## Scope
This document aims to provide background on:
- Why/how The Grid project came to be
- Its intended goals and values
- Differences with the Matrix protocol
- The high-level methodology/processes we want to put in place to ensure this protocol and project are successful
- The set of first priorities that will be worked on
## Origin
As independent developers on the Matrix protocol, we decided to pursue an alternative approach and set of goals and
provide a fresh set of eyes and solutions to the remaining challenges the protocol is trying to solve.
We have spent to this day over 18 months learning, analyzing and reverse-engineering the Matrix protocol, implemented
nearly all components with various use cases, granting us extensive knowledge about its:
- Strengths and weaknesses
- Current limitations
- Security vulnerabilities, both used and hypothetical
While we will like to take an alternate road from the Matrix protocol and become a stand-alone protocol, we believe in
the fundamental concepts and the building blocks and will keep The Grid protocol in line with those for the time being.
## Goals
The protocol will aim for the following goals:
- Solve the problem originally identified by the Matrix protocol, fragmented communications, engaging it from a social
point of view.
- Simple client implementations, relying on the server for heavy lifting
- Federated communication between servers
- Decentralized data storage, spread across participating servers
- Designed for a full self-hostable stack to avoid centralization and/or monopoly
## Values
### Protocol
- Libre (Free) and Open protocol where its contributors and users are empowered
- Privacy of data and metadata
- Security of data and of the network
- Data exchange relying on a federated network, allowing everyone to spin-up a server and provide the service to a set of users
- Decentralized data storage, ensuring no single point of failure
### Management
- Transparency and openness around any talk, design session, workshop, meetings and any other related activity for the
protocol, without secretive or confidential sessions
- Clear, understandable specification documents are the top priority and the main deliverable. A protocol is nothing
without a specification.
- Taken actions and their consequences are used in evaluation, instead of intent.
- Decisions are backed with documented facts/feedback/issues
### Documentation
Documentation will be considered a generic term. Complexity, size, format and shape will be scoped to what is deemed
appropriate by the Governing Body for the various use cases.
As a protocol only exists in its specification and the only seeked resource, documentation will hold a special place for
this project and acts as the currency in spirit of the project, with various value depending on its complexity, size,
format, shape, use case and scope at hand.
It will also be used to enforce various good practices and force facts checking.
More details will be given in a separate, dedicated document to Documentation.
## The Grid vs Matrix
### Differences
#### Core values
Matrix’s ideals are Openness, Decentralization and Encryption [1](#links-and-sources) while The Grid’s ideals are
Freedom, Privacy and Security.
We see Matrix ideals as a subset of The Grid: While Matrix attempts to only solve the technical problem of fragmented
communications, The Grid wants to solve the problem at a social level, a higher level which includes the technical one.
While Matrix values and licenses align with the ideals of Open Source, The Grid would follow those of Libre (Free) software.
This would ensure that users and their rights are the focus and put at the center of each decision, instead of limiting
the discussion to technical matters only.
We believe in a social gain with The Grid protocol rather than technical or monetary gain, which should be reflected in
its own ecosystem.
#### Actors
Given the social purpose of the protocol, we believe involvement, participations and decision making should reflect the
social nature of the protocol.
Therefore, we want to put in place an ecosystem where anyone has a mean to express their opinion, provide feedback, or
contribute to the project itself. Making sure that no single entity is in a position of power alone. Instead, several
entities are in place to keep each other in check.
The creators of this protocol will initially stand alone in a position of power to bootstrap the ecosystem. Afterwards,
the newly-created non-profit will provide within its articles the requirements to join the various boards and decision
groups. See the Governing body section for more details.
#### Documentation
Matrix relies on reverse-engineering reference implementations since its creation 4 years ago, and has yet to produce a
version of its documentation that can be used to implement a full set of usable and interportable implementations.
A new spec proposal process was put in place but we believe it does not address the existing issues that led to the
current situation.
The Grid will therefore take a different approach and attempt to turn documentation into something positive and rewarding
for contributors and implementers, while putting processes and limits to avoid stagnation and status-quo.
It will also take the approach that the spec (the end-result) is to be implemented by at least two projects, independent
of each other, to ensure that the documentation and the specification are not academic exercises, but real-world and
This practical approach where contributors/implementers can see their work be used and recognized, is an example of a
rewarding experience for those who wish to contribute to the project.
### Which is better?
We don’t believe one is better than the other - The Grid and Matrix will each follow different values and a different
type of management.
While they will most likely solve technical problems in different ways, The Grid will initially build from what Matrix
is today and re-use existing and validated documentation. APIs like Client, Application services and Identity service
will remain compatible in the short term as the focus will be on producing a usable document that describes the server
protocol, which Matrix currently doesn’t have.
As The Grid and Matrix will both have concepts of Application Services and therefore Bridges, linking both protocols and
network will be a property that will be heavily leveraged on The Grid side to smooth out any transition or API change
that would make the two protocols otherwise incompatible and unable to exchange data.
If you believe in the same values and ideas as we do, we’ll be happy to have you help us with The Grid.
## Next steps
The high level steps are:
- Creation of a non-profit to kick-start, grow, foster and maintain The Grid protocol.
- Document all the concepts and network exchanges.
- Ensuring that at least two independent implementations validate the documentation.
### Governing body
To ensure the protocol is free of control from a single entity and that the users are empowered to control it, a non-profit
organisation will be created to represent the protocol lead and will be funded on donations.
This body will be created as soon as possible as we believe it is key to ensure neutrality in a communication protocol.
This governing body will be created using public and community-reviewed articles and will put in place technical board(s)
composed of key entities from the community to fulfill its mission.
This governing body will have for mission to ensure the protocol fostering and that its management values are respected
and followed.
Details about this controlling body, its missions and its proposed articles for creation are available in [a separate
### Initial Documentation
We recognize the hard work, effort and engineering level of the current Matrix protocol and wish to build on that,
instead of starting from scratch. We also believe that having a documentation of the as-is is key for people to contribute
constructively and avoid an unfair amount of work spent on reverse-engineering implementations to reach a level of
understanding sufficient to even start meaningful discussions/contributions to improve the protocol.
The biggest lack of documentation is currently the as-is for the server protocol, which deals with internal structures
of a Homeserver, the federation exchanges, the DAG model and its underlying algorithms. It is therefore not possible
to build on an existing documentation for those core concepts.
As the current federation protocol has several critical security vulnerabilities like state resets which have been used
in attacks [2](#links-and-sources) [3](#links-and-sources) on the live network, and very few underlying algorithms and
endpoints have been documented, we will take this opportunity to attempt creating internal structures and federation
exchanges that are immune to those known vulnerabilities, while holding back on any kind of optimization that would
require re-engineering on working, secure and tested concepts.
As all this will be done in the open, as per our core values, current issues to be addressed will be documented and
labeled appropriately so anyone in the community can offer a documented technical solution for them. We believe that
fresh sets of eyes and brains are required to solve those vulnerabilities.
Work has already been started on this and is currently being committed to this Github repository, and a generated online
version is also available. This work will be relocated under the newly The Grid organisation repositories very soon.
### Independent implementations
Meanwhile, we will encourage, foster and proactively seek that independent entities implement the protocol using its
latest documentation.
We believe that a communication protocol that does not have at least two independent implementations is fundamentally
dysfunctional, as its contributors and governing body have failed to communicate to or be understood by other parties
about its worth, merit and the related technical knowledge.
In the near and middle term, emphasis will be put on the various protocol efforts to not diverge too much too fast from
the Matrix protocol. Independent implementers will be able to easily and painlessly implement The Grid as well, ideally
in an invisible manner for the end user.
## Links and Sources
[1]: - “Support Matrix Today” section