Commit Graph

74 Commits

Author SHA1 Message Date
922837dc96 Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
20b80c62ff Change auto-deletion throttling constants to better scale with server size (#23320) 2023-02-23 16:52:40 +01:00
717683d1c3 Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
21bf326356 Autofix Rubocop Rails/Pluck (#23730) 2023-02-20 02:28:40 +01:00
2177daeae9 Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
9edefc779f Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#23318)
* Fix `UserCleanupScheduler` crash when an unconfirmed account has a moderation note

* Add tests
2023-02-07 01:14:44 +01:00
596923da4a Fix typos in source documentation (#21046)
Fixed 2 source comment/documentation typos
2022-12-15 15:57:26 +01:00
5333447be0 Change account deletion requests to spread out over time (#20222) 2022-11-09 14:08:19 +01:00
d0c9ac3919 Fix indexing scheduler trying to index when Elasticsearch is disabled (#19805)
Fix #19646
2022-11-05 22:31:52 +01:00
03d9618595 Fix UserCleanupScheduler crash when an unconfirmed account has a moderation note (#19629)
Fixes #19109
2022-11-01 12:59:23 +01:00
bf0ab3e0fa Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
5c9abdeff1 Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
0b3e4fd5de Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
25e076505e ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
679b7158e3 Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
f6d35ed57d Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
3917353645 Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
3349cbeb3e Fix clean_discarded_statuses (#17822) 2022-03-19 09:24:08 +01:00
50ea54b3ed Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
3d60708508 Fix crash in EmailDomainBlockRefreshScheduler (#17649) 2022-02-26 16:28:28 +01:00
27965ce5ed Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
a29a982eaa Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
35850f8195 Fix localization of cold-start follow recommendations (#17479) 2022-02-08 01:53:49 +01:00
52c1b86964 Fix Ruby 2.5 incompatibility (#17465) 2022-02-07 19:57:06 +01:00
03d59340da Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
14f436c457 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
8e84ebf0cb Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
bda8e4f815 Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
6e50134a42 Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
4ac78e2a06 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
d174d12c83 Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
d9ae3db8d5 Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
f7117646af Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
e89e976e92 Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)
Remove locks from scheduled jobs
2021-03-15 11:17:43 +01:00
3249d35bdc Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
216b85b053 Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
5e1364c448 Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
ed099d8bdc Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
5d8398c8b8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
9014367bd8 Fix background jobs not using locks like they are supposed to (#13361)
Also:

- Fix locks not being removed when jobs go to the dead job queue
- Add UI for managing locks to the Sidekiq dashboard
- Remove unused Sidekiq workers

Fix #13349
2020-03-31 21:59:03 +02:00
401f32f9ee Fix expired announcements being re-published (#12964) 2020-01-26 22:43:18 +01:00
f52c988e12 Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
c21386cff5 Improve IP cleanup query (#11871) 2019-09-17 08:44:25 +02:00
70ddef2654 Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
0d80f686d8 Add tootctl preview_cards remove (#11320)
* Add `tootctl preview_cards remove`

* fix code style

* Remove `Scheduler::PreviewCardsCleanupScheduler` file

* fix code style again
Add exclude case where image_file_name is blank

* Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-28 13:48:19 +02:00
bd87e66679 Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
b3f44aa186 Add periodic removal of older thumbnails for preview cards (#11304) 2019-07-15 07:50:14 +02:00
23aeef52cc Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
bcfff65195 Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
8ad57a0426 Fix PublishScheduledStatusWorker being queued with no arguments (#9724) 2019-01-06 09:21:16 +01:00