Commit Graph

85 Commits

Author SHA1 Message Date
Eugen Rochko
a0fad5c8bb Change indexing frequency from 5 minutes to 1 minute, add locks to schedulers (#26304) 2023-08-03 11:04:05 +02:00
Claire
b4e739ff0f Change interaction modal in web UI (#26075)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2023-07-27 16:11:17 +02:00
Claire
144a406d33 Clean up unused application records (#24871) 2023-07-21 13:13:16 +02:00
Emelia Smith
3a91603b15 Prevent UserCleanupScheduler from overwhelming streaming (#25519) 2023-06-20 18:04:35 +02:00
Claire
c3c7d59072 Fix infinite loop in AccountsStatusesCleanupScheduler (#24840) 2023-05-04 10:13:00 +02:00
Claire
8b636a29c6 Improve automatic post cleanup worker performances (#24785) 2023-05-03 10:31:40 +02:00
Claire
32a030dd74 Rewrite import feature (#21054) 2023-05-02 12:08:48 +02:00
Claire
5dc3173ef8 Fix AccountsStatusesCleanupScheduler not spreading deletes across accounts correctly (#24607) 2023-04-23 22:25:40 +02:00
Claire
501d6197c4 Change automatic post deletion thresholds and load detection (#24614) 2023-04-21 18:14:19 +02:00
Vyr Cossont
d5ad8b6422 Fix Redis client and type errors introduced in #24285 (#24342) 2023-04-01 08:28:35 +02:00
Vyr Cossont
7646ad8a2b IndexingScheduler: fetch and import in batches (#24285)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-03-31 14:38:47 +02:00
Jean byroot Boussier
922837dc96 Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
Claire
20b80c62ff Change auto-deletion throttling constants to better scale with server size (#23320) 2023-02-23 16:52:40 +01:00
Nick Schonning
717683d1c3 Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
Nick Schonning
21bf326356 Autofix Rubocop Rails/Pluck (#23730) 2023-02-20 02:28:40 +01:00
Nick Schonning
2177daeae9 Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
Claire
9edefc779f Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#23318)
* Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note

* Add tests
2023-02-07 01:14:44 +01:00
luzpaz
596923da4a Fix typos in source documentation (#21046)
Fixed 2 source comment/documentation typos
2022-12-15 15:57:26 +01:00
Claire
5333447be0 Change account deletion requests to spread out over time (#20222) 2022-11-09 14:08:19 +01:00
Eugen Rochko
d0c9ac3919 Fix indexing scheduler trying to index when Elasticsearch is disabled (#19805)
Fix #19646
2022-11-05 22:31:52 +01:00
Claire
03d9618595 Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#19629)
Fixes #19109
2022-11-01 12:59:23 +01:00
Eugen Rochko
bf0ab3e0fa Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
Eugen Rochko
5c9abdeff1 Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Eugen Rochko
0b3e4fd5de Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
Jeremy Kescher
25e076505e ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
Eugen Rochko
679b7158e3 Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Eugen Rochko
f6d35ed57d Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
Eugen Rochko
3917353645 Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Jeong Arm
3349cbeb3e Fix clean_discarded_statuses (#17822) 2022-03-19 09:24:08 +01:00
Eugen Rochko
50ea54b3ed Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Claire
3d60708508 Fix crash in EmailDomainBlockRefreshScheduler (#17649) 2022-02-26 16:28:28 +01:00
Eugen Rochko
27965ce5ed Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko
a29a982eaa Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko
35850f8195 Fix localization of cold-start follow recommendations (#17479) 2022-02-08 01:53:49 +01:00
Claire
52c1b86964 Fix Ruby 2.5 incompatibility (#17465) 2022-02-07 19:57:06 +01:00
Claire
03d59340da Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Eugen Rochko
14f436c457 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Eugen Rochko
8e84ebf0cb Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Takeshi Umeda
bda8e4f815 Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
Eugen Rochko
6e50134a42 Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
4ac78e2a06 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Eugen Rochko
d174d12c83 Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire
d9ae3db8d5 Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko
f7117646af Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko
e89e976e92 Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)
Remove locks from scheduled jobs
2021-03-15 11:17:43 +01:00
ThibG
3249d35bdc Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Eugen Rochko
216b85b053 Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Eugen Rochko
5e1364c448 Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko
ed099d8bdc Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Eugen Rochko
5d8398c8b8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00