Commit graph

1083 commits

Author SHA1 Message Date
Neil Alexander e15f6676ac
Consolidation of roomserver APIs (#994)
* Consolidation of roomserver APIs

* Comment out alias tests for now, they are broken

* Wire AS API into roomserver again

* Roomserver didn't take asAPI param before so return to that

* Prevent roomserver asking AS API for alias info

* Rename some files

* Remove alias_test, incoherent tests and unwanted appservice integration

* Remove FS API inject on syncapi component
2020-05-01 10:48:17 +01:00
Kegsay ebbfc12592
Add tests for the storage interface (#995)
* Move docs to interface

* Add tests around syncing

* Add topology token test

* Linting
2020-04-30 17:15:29 +01:00
Neil Alexander 540f6fcd94 Update gmsl for key validity fix 2020-04-30 13:50:11 +01:00
Neil Alexander 77fe509031
Enable v5 rooms (#992)
* Enable v5 roooms

* Update sytest-whitelist

* Enable v5 rooms by default, update gomatrixserverlib
2020-04-29 19:37:00 +01:00
Kegsay 4ad52c67ca
Honour history_visibility when backfilling (#990)
* Make backfill work for shared history visibility

* fetch missing state on backfill to remember snapshots correctly

* Fix gmsl to not mux in auth events into room state

* Whoops

* Linting
2020-04-29 18:41:45 +01:00
Neil Alexander 458b364781 Update gomatrixserverlib 2020-04-29 15:33:17 +01:00
Neil Alexander 64e94e9a6f
Join room support in federation sender (#989)
* Implement PerformJoinRequest

* Rename perform functions

* Check send join response

* Temporary wiring to test federation sender room joins

* Actually pass through the config

* Make sure membership content shows join
2020-04-29 15:29:39 +01:00
Neil Alexander a308e61331
Federation sender API remodel (#988)
* Define an input API for the federationsender

* Wiring for rooomserver input API and federation sender input API

* Whoops, commit common too

* Merge input API into query API

* Rename FederationSenderQueryAPI to FederationSenderInternalAPI

* Fix dendritejs

* Rename Input to Perform

* Fix a couple of inputs -> performs

* Remove needless storage interface, add comments
2020-04-29 11:34:31 +01:00
Kegsay a4b9edb28e
Dependency inject the federation client so p2p binaries work as expected (#987) 2020-04-28 16:51:16 +01:00
Kegsay 0354836b57
Unbreak the wasm build (#986) 2020-04-28 16:22:00 +01:00
Kegsay 35b7cbd5d8
sql/backwards_extremities: Shift to table format and share code (#985)
* sql/backwards_extremities: Shift to table format and share code

This is an initial cut to reduce boilerplate at the storage layer.
It removes the need for 2x `_table.go` files, one for each DB engine,
replacing it with a single struct which has an interface which
implements the raw SQL statements.

The actual impl sits alongside the interface declaration which is
generally regarded as best practice (though no canonical sources).
Especially in this case where the impl is tiny (functions returning
strings) and relies heavily on the function signatures of the
table struct (for parameters), having the context in the same file
is useful.

* Remove _table redundancy
2020-04-28 15:50:24 +01:00
Neil Alexander 5071ecb8b3
Fix URLDecodeMapValues (#984)
* Update gomatrixserverlib

* Fix URLDecodeMapValues

* Update gomatrixserverlib
2020-04-28 12:50:49 +01:00
Kegsay 6d832ae544
Implement backfill in the roomserver (#983)
* Initial cut for backfilling

The syncserver now asks the roomserver via QueryBackfill (which already
existed to *handle* backfill requests) which then makes federation requests
via gomatrixserverlib.RequestBackfill.

Currently, tests fail on subsequent /messages requests because we don't know
which servers are in the room, because we are unable to get state snapshots
from a backfilled event because that code doesn't exist yet.

* WIP backfill, doesn't work

* Make initial backfill pass checks

* Persist backfilled events with state snapshots

* Remove debug lines

* Linting

* Review comments
2020-04-28 11:46:47 +01:00
Neil Alexander 3a858afca2
Loopback event from invite response (#982)
* Working invite v2 support

* Fix copyright notice

* Update gomatrixserverlib

* Add fallthrough

* Add missing continue

* Update sytest-whitelist, gomatrixserverlib

* Update gomatrixserverlib to test matrix-org/gomatrixserverlib#181

* Update gomatrixserverlib
2020-04-28 10:53:07 +01:00
Neil Alexander 87f05721b0 Update gomatrixserverlib 2020-04-27 15:47:36 +01:00
Neil Alexander 3ab8ebf6b8
More invite support (#979)
* Update gomatixserverlib

* Try to build invite stripped state if not given to us

* SendInvite improvements

* Transpose invite_room_state into invite_state.events for sync API

* Remove syncapi debugging output

* Use RespInviteV2

* Update gomatrixserverlib

* Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?)

* Federation sender use invite_room_state, room server try to insert membership state

* Check supported room versions on the invite endpoint

* Prevent roomserver query API from trying to handle requests for stub rooms

* Adding a nolint

* Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead

* Review comments
2020-04-24 16:30:25 +01:00
Kegan Dougal be558f02aa Add new sytests to list 2020-04-24 10:39:11 +01:00
Kegsay a202d88fe5
Use a single storage.Database interface (#978) 2020-04-24 10:38:58 +01:00
Neil Alexander c30b12b5a1
Fix sarama import URLs (#856)
* Fix sarama import URLs

* Update gomatrixserverlib

* Update naffka

* Update naffka

* Update in kafka-producer
2020-04-22 15:26:56 +01:00
Behouba Manassé 423c656146
Response from /send_join now use gomatrixserverlib.RespSendJoin (#796)
* response from /send_join now use gomatrixserverlib.RespSendJoin

* Update Dendrite gomatrixserverlib version

* Fix spelling

Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
Co-authored-by: Cnly <minecnly@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-22 14:13:11 +01:00
Neil Alexander a466e9e9cc
LRU cache for room versions in RS query API (#976)
* Experimental LRU cache for room versions

* Don't accidentally try to type-assert nil

* Also reduce hits on query API

* Use hashicorp implementation which mutexes for us

* Define const for max cache entries

* Rename to be specifically immutable, panic if we try to mutate a cache entry

* Review comments

* Remove nil guards, give roomserver integration test a cache

* go mod tidy
2020-04-22 13:00:05 +01:00
Neil Alexander 71f9d35b7c Update gomatrixserverlib 2020-04-21 10:04:06 +01:00
Neil Alexander 9045b8e89f
Perspective key fetching, some federation room join fixes (#975)
* Update gomatrixserverlib

* Test matrix.org as perspective key server

* Base64 decode better

* Optional strict validity checking in gmsl

* Update gomatrixserverlib

* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)

* Update gomatrixserverlib, debug logging

* Remove debugging output

* More verbose debugging

* Print outliers

* Increase timeouts for testing, observe contexts before trying to join over more servers

* Don't block on roomserver (experimental)

* Don't block on roomserver

* Update gomatrixserverlib

* Update gomatrixserverlib

* Configurable perspective key fetchers

* Output number of configured keys for perspective

* Example perspective config included

* Undo debug stack trace

* Undo debug stack trace

* Restore original HTTP listener in monolith

* Fix lint

* Review comments

* Set default HTTP server timeout to 5 minutes now, block again when joining

* Don't use HTTP address for HTTPS whoops

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Actually add perspectives

* Actually add perspectives

* Update gomatrixserverlib
2020-04-20 17:42:34 +01:00
Kegsay c62844234b
Add login sytests (#974) 2020-04-20 14:19:25 +01:00
Alex Chen 9f5ece8045
Fix nil http clients provided to components in polylith mode (#973)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-19 11:18:38 +01:00
Alex Chen e1a4ada0e1
Ensure room version tests can be grouped whether failed or passed (#972)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-18 11:01:48 +01:00
Kegsay 0d116d9222
Allow anyone to publish rooms (#971)
* Allow anyone to publish rooms

* Formatting and unused var

* Review comments
2020-04-17 12:17:20 +01:00
Neil Alexander 3c2e6f967b
Federation fixes and error handling (#970)
* Improve error handling in federation /send endpoint a bit

* Remove unknownRoomError, use unmarshalError when unable to get room ID

* Swap out a couple more internal server errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Return bad limit in error

* Same with domain/userid
2020-04-16 17:59:55 +01:00
Neil Alexander 3110a81996
Set default room version to 4 (#957)
* Set default room version to 4

* Default to v1 when room_version key missing

Signed-off-by: Alex Chen <minecnly@gmail.com>

* Squashed commit of the following:

commit c1bca95adb
Author: Kegsay <kegan@matrix.org>
Date:   Thu Apr 16 10:06:55 2020 +0100

    Add SQL tracing via DENDRITE_TRACE_SQL (#968)

    * Add SQL tracing via DENDRITE_TRACE_SQL

    Add this to `internal/sqlutil` in preparation for #897

    * Not entirely

commit c2ea961909
Author: Kegsay <kegan@matrix.org>
Date:   Wed Apr 15 17:48:40 2020 +0100

    Add HTTP trace logging (#965)

    * Dump all requests/response server-side

    * Wrap in DENDRITE_TRACE

    * DENDRITE_TRACE_HTTP is better

    * Bugfix for response body and linting

    * False is true and true is false

    * Linting

    * How did this get missed

    * More linting

* Update gomatrixserverlib

* Remove unneeded imports

Co-authored-by: Cnly <minecnly@gmail.com>
2020-04-16 12:53:27 +01:00
Kegsay c1bca95adb
Add SQL tracing via DENDRITE_TRACE_SQL (#968)
* Add SQL tracing via DENDRITE_TRACE_SQL

Add this to `internal/sqlutil` in preparation for #897

* Not entirely
2020-04-16 10:06:55 +01:00
Kegsay c2ea961909
Add HTTP trace logging (#965)
* Dump all requests/response server-side

* Wrap in DENDRITE_TRACE

* DENDRITE_TRACE_HTTP is better

* Bugfix for response body and linting

* False is true and true is false

* Linting

* How did this get missed

* More linting
2020-04-15 17:48:40 +01:00
Neil Alexander dadb06f6ad
Use topological ordering for /messages response (#966)
* Use topological ordering for /messages response

* Update gomatrixserverlib
2020-04-15 16:10:18 +01:00
Neil Alexander 7b3edf4622
Tweaks to backfill (#964) 2020-04-15 14:39:27 +01:00
Neil Alexander 895a72b6ee
Move /room/{roomID}/state endpoints into client API (#606) (#962)
* Move /room/{roomID}/state endpoints into client API (#606)

* Update sytest-whitelist

* Blacklist tests which rely on endpoints we don't implement
2020-04-14 18:36:08 +01:00
Hilmar Gústafsson 73d2f59e30
WIP: Add libp2p-go (#956)
* Add libp2p-go

* Some tweaks, tidying up

(cherry picked from commit 1a5bb121f8121c4f68a27abbf25a9a35a1b7c63e)

* Move p2p dockerfile

(cherry picked from commit 8d3bf44ea1bf37f950034e73bcdc315afdabe79a)

* Remove containsBackwardsExtremity

* Fix some linter errors, update some libp2p packages/calls, other tidying up

* Add -port for dendrite-p2p-demo

* Use instance name as key ID

* Remove P2P demo docker stuff, no longer needed now that we have SQLite

* Remove Dockerfile-p2p too

* Remove p2p logic from dendrite-monolith-server

* Inject publicRoomsDB in publicroomsapi

Inject publicRoomsDB instead of switching on base.libP2P.
See: https://github.com/matrix-org/dendrite/pull/956/files?file-filters%5B%5D=.go#r406276914

* Fix lint warning

* Extract mDNSListener from base.go

* Extract CreateFederationClient into demo

* Create P2PDendrite from BaseDendrite

Extract logic specific to P2PDendrite from base.go

* Set base.go to upstream/master

* Move pubsub to demo cmd

* Move PostgreswithDHT to cmd

* Remove unstable features

* Add copyrights

* Move libp2pvalidator into p2pdendrite

* Rename dendrite-p2p-demo -> dendrite-demo-libp2p

* Update copyrights

* go mod tidy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 16:15:59 +01:00
Kegsay 48303d06cb
Are we synapse yet? (#960)
* Add a script which tracks synapse parity based on sytests

Processes results.tap to give you a percentage check.

* Pretty

* argparse and -v output
2020-04-14 16:05:15 +01:00
Kegsay 609f034bfb
S7evinK: basicauth metrics (#961)
* Add setting to enable/disable metrics (#461)
Add basic auth to /metric handlers

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add warning message if metrics are exposed without protection

* Remove redundant type conversion

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* SetBasicAuth per test case

* Update warning message and change loglevel to warn

* Update common/config/config.go

* Update dendrite-config.yaml

Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 15:54:35 +01:00
mohit kumar singh 2c43e222bd
support non-exclusive namespaces for AS (#828)
Signed-off-by: MohitKS5 <mohitkumarsingh907@gmail.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2020-04-14 15:31:27 +01:00
Prateek Sachan 1321f8da80
Check if user has the power level to edit the room visibility (#900)
* Check if user has the power level to edit the room visibility

* fix review changes

* Add nil check for queryEventsRes.StateEvents

Co-authored-by: Alex Chen <Cnly@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 14:07:03 +01:00
Alex Chen d359851708
Update guides for running sytest (#958)
* Update guides for running sytest

Bring the details for running sytest against postgres up to date; prefer docker over manual setup

Signed-off-by: Alex Chen <minecnly@gmail.com>

* Better flags

Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-12 13:48:24 +01:00
Prateek Sachan 317658acea
Added checks for JSON body in accounts_data endpoint (#863)
Signed-off-by: Prateek Sachan <psachan@cs.iitr.ac.in>
2020-04-12 00:47:05 +08:00
Neil Alexander dacee648f7
Federation for v3/v4 rooms (#954)
* Update gomatrixserverlib

* Default to room version 4

* Update gomatrixserverlib

* Limit prev_events and auth_events

* Fix auth_events, prev_events

* Fix linter issues

* Update gomatrixserverlib

* Fix getState

* Update sytest-whitelist

* Squashed commit of the following:

commit 067b875063
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Apr 3 14:29:06 2020 +0100

    Invites v2 endpoint (#952)

    * Start converting v1 invite endpoint to v2

    * Update gomatrixserverlib

    * Early federationsender code for sending invites

    * Sending invites sorta happens now

    * Populate invite request with stripped state

    * Remodel a bit, don't reflect received invites

    * Handle invite_room_state

    * Handle room versions a bit better

    * Update gomatrixserverlib

    * Tweak order in destinationQueue.next

    * Revert check in processMessage

    * Tweak federation sender destination queue code a bit

    * Add comments

commit 955244c092
Author: Ben B <benne@klimlive.de>
Date:   Fri Apr 3 12:40:50 2020 +0200

    use custom http client instead of the http DefaultClient (#823)

    This commit replaces the default client from the http lib with a custom one.
    The previously used default client doesn't come with a timeout. This could cause
    unwanted locks.
    That solution chosen here creates a http client in the base component dendrite
    with a constant timeout of 30 seconds. If it should be necessary to overwrite
    this, we could include the timeout in the dendrite configuration.
    Here it would be a good idea to extend the type "Address" by a timeout and
    create an http client for each service.

    Closes #820

    Signed-off-by: Benedikt Bongartz <benne@klimlive.de>

    Co-authored-by: Kegsay <kegan@matrix.org>

* Update sytest-whitelist, sytest-blacklist

* Update go.mod/go.sum

* Add some error wrapping for debug

* Add a NOTSPEC to common/events.go

* Perform state resolution at send_join

* Set default room version to v2 again

* Tweak GetCapabilities

* Add comments to ResolveConflictsAdhoc

* Update sytest-blacklist

* go mod tidy

* Update sytest-whitelist, sytest-blacklist

* Update versions

* Updates from review comments

* Update sytest-blacklist, sytest-whitelist

* Check room versions compatible at make_join, add some comments, update gomatrixserverlib, other tweaks

* Set default room version back to v2

* Update gomatrixserverlib, sytest-whitelist
2020-04-09 15:46:06 +01:00
Neil Alexander 067b875063
Invites v2 endpoint (#952)
* Start converting v1 invite endpoint to v2

* Update gomatrixserverlib

* Early federationsender code for sending invites

* Sending invites sorta happens now

* Populate invite request with stripped state

* Remodel a bit, don't reflect received invites

* Handle invite_room_state

* Handle room versions a bit better

* Update gomatrixserverlib

* Tweak order in destinationQueue.next

* Revert check in processMessage

* Tweak federation sender destination queue code a bit

* Add comments
2020-04-03 14:29:06 +01:00
Ben B 955244c092
use custom http client instead of the http DefaultClient (#823)
This commit replaces the default client from the http lib with a custom one.
The previously used default client doesn't come with a timeout. This could cause
unwanted locks.
That solution chosen here creates a http client in the base component dendrite
with a constant timeout of 30 seconds. If it should be necessary to overwrite
this, we could include the timeout in the dendrite configuration.
Here it would be a good idea to extend the type "Address" by a timeout and
create an http client for each service.

Closes #820

Signed-off-by: Benedikt Bongartz <benne@klimlive.de>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-04-03 11:40:50 +01:00
Alex Chen 2c8950221e
Fix: show-expected-fail-tests.sh ignores # in whitelist and strips quotes in test names in output (#943)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-03-31 15:53:09 +01:00
Kegsay 8fbe9f4078
Implement typing over federation (#949)
Also fix a pet peeve of mine: not putting units on things!!!

Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30 16:40:28 +01:00
Kegsay 11a8059bba
Rename the typing server to EDU server (#948)
* Blunt move and sed rename

* Rename common/ refs to typing

* Rename internal stuff in eduserver

* Rename docs and scripts

* Rename constants/filenames, goimports everything to re-order imports
2020-03-30 15:02:20 +01:00
Kegan Dougal f72b759426 Update to latest go-sqite-js 2020-03-30 12:47:19 +01:00
Neil Alexander 664f31ec98 Ensure state res results are unique 2020-03-30 09:51:45 +01:00
Neil Alexander 05e1ae8745
Further room version wiring (#936)
* Room version 2 by default, other wiring updates, update gomatrixserverlib

* Fix nil pointer exception

* Fix some more nil pointer exceptions hopefully

* Update gomatrixserverlib

* Send all room versions when joining, not just stable ones

* Remove room version cquery

* Get room version when getting events from the roomserver database

* Reset default back to room version 2

* Don't generate event IDs unless needed

* Revert "Remove room version cquery"

This reverts commit a170d58733.

* Query room version in federation API, client API as needed

* Improvements to make_join send_join dance

* Make room server producers use headered events

* Lint tweaks

* Update gomatrixserverlib

* Versioned SendJoin

* Query room version in syncapi backfill

* Handle transaction marshalling/unmarshalling within Dendrite

* Sorta fix federation (kinda)

* whoops commit federation API too

* Use NewEventFromTrustedJSON when getting events from the database

* Update gomatrixserverlib

* Strip headers on federationapi endpoints

* Fix bug in clientapi profile room version query

* Update gomatrixserverlib

* Return more useful error if room version query doesn't find the room

* Update gomatrixserverlib

* Update gomatrixserverlib

* Maybe fix federation

* Fix formatting directive

* Update sytest whitelist and blacklist

* Temporarily disable room versions 3 and 4 until gmsl is fixed

* Fix count of EDUs in logging

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Rely on EventBuilder in gmsl to generate the event IDs for us

* Some review comments fixed

* Move function out of common and into gmsl

* Comment in federationsender destinationqueue

* Update gomatrixserverlib
2020-03-27 16:28:22 +00:00