Commit graph

51 commits

Author SHA1 Message Date
Neil Alexander 39e8d1cc6f
Track knocking in membership updater (#1935)
* Topologically sort outliers in SendEventWithState

* Knock in membership updater

* Update gomatrixserverlib

* Update gomatrixserverlib

* Get the NID of the knock event properly for the membership updater
2021-07-22 12:26:58 +01:00
Neil Alexander c1447a58e5
Various alias fixes (#1934)
* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments
2021-07-21 16:53:50 +01:00
Neil Alexander f0f8c7f055
Optimise QueryServerJoinedToRoom (#1933)
* Optimise checking if a server is in a room

* Fix queries

* Fix queries
2021-07-21 13:06:32 +01:00
David Spenler 8d8fe485b4
Fix failing ban tests (#1884)
* Add room membership and powerlevel checks for func SendBan

* Added non-error return to func GetStateEvent when no state events with the specified state key are found

* Add passing tests to whitelist

* Fixed formatting

* Update roomserver/storage/shared/storage.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: kegsay <kegsay@gmail.com>
2021-07-19 18:33:05 +01:00
Neil Alexander c8408a6387
Add more optimised code path for checking if we're in a room (#1909)
* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership
2021-07-09 16:36:45 +01:00
Neil Alexander 5357df36c9
Fix panic in roomserver 2021-06-21 09:41:12 +01:00
Neil Alexander 5ce1fe80de
State storage refactor (#1839)
* Hash-deduplicated state storage (and migrations) for PostgreSQL and SQLite

* Refactor droomserver database setup for migrations

* Fix conflict statements

* Update migration names

* Set a boundary for old to new block/snapshot IDs so we don't rewrite them more than once accidentally

* Create sequence if not exists

* Fix boundary queries

* Fix boundary queries

* Use Query

* Break out queries a bit

* More sequence tweaks

* Query parameters are not playing the game

* Injection escaping may not work for CREATE SEQUENCE after all

* Fix snapshot sequence name

* Use boundaried IDs in SQLite too

* Use IFNULL for SQLite

* Use COALESCE in PostgreSQL

* Review comments @Kegsay
2021-04-26 13:25:57 +01:00
Neil Alexander d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander 9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander 1ce9c52442
Don't recalculate event IDs so often (#1610)
* Don't recalculate event IDs so often

* Revert invite change

* Make sure we're using the right NIDs

* Update gomatrixserverlib

* Update to NewEventFromTrustedJSONWithEventID

* Fix go.mod

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243

* Use BulkSelectEventID
2020-12-04 10:41:07 +00:00
Neil Alexander 20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
S7evinK eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Kegsay 6b8791b868
Always call overridden form of GetLatestEventsForUpdate (#1554)
This ensures we don't make txns on sqlite still, which can cause
'database is locked' errors.
2020-10-20 19:32:33 +01:00
Kegsay eb86e2b336
Fix sqlite locking bugs present on sytest (#1543)
* Fix sqite locking bugs present on sytest

Comments do the explaining.

* Fix deadlock in sqlite mode

Caused by starting a writer whilst within a writer

* Only complain about invalid state deltas for non-overwrite events

* Do not re-process outlier unnecessarily
2020-10-20 11:42:54 +01:00
Neil Alexander e154c45b51
Better logging around db.StoreEvent 2020-10-15 14:14:17 +01:00
Kegsay e3c2b081c7
txn nil guard when rolling back LatestEventsUpdater (#1524)
* txn nil guard when rolling back LatestEventsUpdater

* Spell lint correctly
2020-10-14 17:05:09 +01:00
Kegsay d7ea814fa8
Wrap NewMembershipUpdater in a db writer (#1515) 2020-10-13 10:20:27 +01:00
Neil Alexander d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander 45de9dc1c0
Use room version cache in Events() 2020-09-21 16:49:37 +01:00
Kegsay 18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Matthew Hodgson 39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander 35564dd73c
Process membership updates in writers (#1414) 2020-09-08 17:48:07 +01:00
Kegsay 7913759921
Remove QueryBulkStateContent from current state server (#1404)
* Remove QueryBulkStateContent from current state server

Expected fail due to db impl not existing

* Implement query bulk state content

* Fix up rejecting invites over federation

* Fix bulk content marshalling
2020-09-07 12:38:09 +01:00
Neil Alexander b9caccbce8
Update GetStateEvent behaviour (#1399) 2020-09-04 19:40:21 +01:00
Kegsay ca8dcf46b7
Remove QuerySharedUsers from current state server (#1396)
* Remove QuerySharedUsers from current state server

* Bugfixes
2020-09-04 14:25:01 +01:00
Kegsay 33b8143a95
Implement more CSS storage functions in roomserver (#1388) 2020-09-03 18:27:02 +01:00
Kegsay b20386123e
Move currentstateserver API to roomserver (#1387)
* Move currentstateserver API to roomserver

Stub out DB functions for now, nothing uses the roomserver version yet.

* Allow it to startup

* Implement some current-state-server storage interface functions

* Add missing package
2020-09-03 17:20:54 +01:00
Kegsay 02a73f29f8
Expand RoomInfo to cover more DB storage functions (#1377)
* Factor more things to RoomInfo

* Factor out remaining bits for RoomInfo

* Linting for now
2020-09-02 10:02:48 +01:00
Kegsay 6d79f04354
Add RoomInfo metadata struct (#1367)
* Add RoomInfo struct

* Remove RoomNID and replace with RoomInfo

* Bugfix and remove another needless query

* nil guard
2020-09-01 12:40:49 +01:00
Neil Alexander 819b926820
Make redactions permanent (#1337)
* Make redactions permanent

* Update storage.go
2020-08-25 15:44:19 +01:00
Neil Alexander c8b873abc8
Roomserver NID caches (#1335)
* Initial work on roomserver NID caches

* Give caches to roomserver storage

* Populate caches

* Fix bugs

* Fix WASM build

* Don't hit cache twice in RoomNIDExcludingStubs

* Store reverse room ID-room NID mapping, consult caches when assigning NIDs
2020-08-25 12:32:29 +01:00
Neil Alexander 9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander 068a3d3c9f
Roomserver per-room input parallelisation (Postgres) (#1289)
* Per-room input mutex

* GetMembership should use transaction when assigning state key NID

* Actually use writer transactions rather than ignoring them

* Limit per-room mutexes to Postgres

* Flip the check in InputRoomEvents
2020-08-20 16:24:33 +01:00
Neil Alexander 5ad47d3b3d
Fix more roomserver transactions/locks (#1287)
* Fix transaction to InsertTransaction

* Remove unnecessary txn, add txns around setting up updaters
2020-08-20 09:24:52 +01:00
Neil Alexander b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Neil Alexander 775b04d776
Roomserver updater changes (#1283)
* Take input transaction when setting up updaters

* Fix nil pointer exceptions

* Rename room recent events updater to latest events updater

* Contd rename room recent events updater to latest events updater

* Remove unnecessary interfaces for latest events and membership updaters
2020-08-19 13:24:54 +01:00
Neil Alexander f54689f862
Fix lint error 2020-07-21 17:21:48 +01:00
Neil Alexander b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Kegsay d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Kegsay 99ea1f9b48
Emit redacted_event from the roomserver when redactions are validated (#1186)
* Emit redacted_event from the roomserver when redactions are validated

- Consume them in the currentstateserver and act accordingly.
- Add integration test for the roomserver to check that injecting
  `m.room.redaction` events result in `redacted_event` being emitted.

* Linting

* Ignore events that redact themselves
2020-07-07 12:51:55 +01:00
Kegsay 52c77cb4b4
Implement core redaction logic (#1185)
* Implement core redaction logic

- Add a new `redactions_table.go` which tracks the mapping of
  the redaction event ID and the redacted event ID
- Mark redactions as 'validated' when we have both events.
- When redactions are validated, add `unsigned.redacted_because`
  and modify the `eventJSON` accordingly.

Note: We currently do NOT redact the event content - it's gated
behind a feature flag - until we have tested redactions a bit more.

* Linting
2020-07-06 17:49:15 +01:00
Kegsay 4c1e6597c0
Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver (#1174)
* Use content_value instead of membership

* Fix build

* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver

- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
  which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
    * Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
    * Apply pagination parameters to the slice.
    * Do a `QueryBulkStateContent` request to the currentstateserver to pull out
      required state event *content* (not entire events).
    * Aggregate and return the chunk.

Mostly but not fully implemented (DB queries on currentstateserver are missing)

* Fix pq query

* Make postgres work

* Make sqlite work

* Fix tests

* Unbreak pagination tests

* Linting
2020-07-02 15:41:18 +01:00
Kegsay 4897beabee
Finish implementing retiring invites (#1166)
* Pass retired invites to the syncapi with the event ID of the invite

* Implement retire invite streaming

* Update whitelist
2020-06-26 11:07:52 +01:00
Kegsay ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Kegsay a6f995eb45
Merge Updater structs (#1069)
* Move Updater structs to shared and use it for postgres

* Add constructors for NewXXXUpdater and a useTxns flag

In sqlite, we set useTxns=false and comment why.

* Handle nil txn

* Handle nil in transaction

* Missed one

* Close the txn at the right time

* Don't close the transaction as we reuse it between calls
2020-05-28 11:15:21 +01:00
Kegsay a7f12bce79
Convert remaining roomserver tables (#1065)
* Convert invites table

* Convert membership table

* Factor out remaining functions except for *Updater structs

* Remove 'implements' comments from long-forgotten interfaces. Move those comments to storage.Database
2020-05-27 11:03:47 +01:00
Kegsay c0c5d9452a Convert room_aliases previous_events state_block and state_snapshot tables (#1064)
* Convert state_snapshot and state_block tables

* Convert room_aliases and previous_events tables

* Add missing table
2020-05-27 09:36:09 +01:00
Kegsay 19aa44ecae
Convert transactions/rooms table to share more code (#1063)
* Convert rooms table

* Convert transactions table

* Convert rooms table and factor out lots of functions

* I think you'll be needing this..
2020-05-26 18:23:39 +01:00
Kegsay 803af87dc4
Convert events/event_json tables to share code (#1062)
* Convert event_json table

* Convert the events table
2020-05-26 16:45:28 +01:00