Commit Graph

634 Commits (master)

Author SHA1 Message Date
Anna-Maria Meriniemi 4fe0ff5b1a
Optimize magnet generation (#594)
with our trusty tool: caching!
2020-07-12 00:14:35 -07:00
Anna-Maria Meriniemi e545f8ae19
Introduce baked queries (#592)
SQA's baked queries prepares the queries in advance, caching yada yada.
Makes thing a bit faster.
Also bigger speedup included is a shoddy cache for the total torrent
count (only applied to baked queries currently). Caching the value for a
few dozen seconds shaves off some wasted time, as it's mostly just used
for pagination.
2020-07-12 00:14:06 -07:00
queue 4fcef92b94
elasticsearch 7.x compatability (#576)
* es_mapping: update turning off dynamic mappings

they changed it in 6.x

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
https://github.com/elastic/elasticsearch/pull/25734

* es_mapping: remove _all field

deprecated in 6.0 anyway

* es_mapping.yml: fix deprecated mapping type

https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html#_schedule_for_removal_of_mapping_types

it gives a really unhelpful error otherwise, oof.

* es: fix remaining 7.xisms

the enabled: false apparently only applies to
"object" fields now, need index: false

and the _type got removed everywhere. Seems to work now.

* Fix weird offset error with word_delimiter_graph

yet another es7-ism i guess

* Fix warning and some app stuff for ES 7.x

Co-authored-by: Arylide <Arylide@users.noreply.github.com>
2020-07-12 00:10:47 -07:00
Nicolas F 72087ddaaf ratelimit: add app config examples 2020-06-23 00:49:26 +02:00
Nicolas F 5c943f35e3 Add login endpoint rate limiting
This doesn't discriminate between failed logins and successful
logins, but only counts POST requests. The limit is set to 6 per
hour.
2020-06-23 00:49:26 +02:00
Nicolas F 611f0c5706 search: allow admins to ignore the max page limit (#589) 2019-10-29 19:16:14 -07:00
Nicolas F 5da7635164 account: add IP rate limiting for account creation
Add a configurable cooldown in seconds for which no further accounts
can be created from the same IP.
2019-08-26 10:10:11 +02:00
Nicolas F 97c32a483f backend: count IP uploads in the user ratelimit (#583)
Users could double their ratelimit by uploading some torrents as
anonymous submissions, then log into their account and post more.

We can stop this by making the filter_uploader helper function use
an sqlalchemy.or_ query to check for uploads from either that user
or that user's IP.
2019-08-22 14:22:12 +02:00
A nyaa developer 532439356f nuke comments (#577)
* nuke comments

* improve update_comment_count_db
2019-08-13 13:15:16 -07:00
Nicolas F 80c9d580eb Improve cache configuration (#564)
The Flask-Caching cache can now properly be configured from the
config.py, and redis caching has experimentally been tested and
confirmed to be working in theory.

We also document that one may want to use CACHE_THRESHOLD to limit
the maximum number of items in the simple cache.
2019-08-10 18:53:57 -07:00
Nicolas F d8e796f3e0 Move to Python 3.7 and update dependencies (#551)
* forms: replace re._pattern_type with re.Pattern

Python 3.7 removed re._pattern_type and replaced it with
re.Pattern.

* readme: update for Python 3.7

* Update requirements

Also remove some unused ones which were neither a direct dependency
nor a dependency of our dependencies.

* account: force ASCII usernames on login form

Our database doesn't like it when we check for unicode data in
a column that stores ASCII data, so let's stop it before it
gets that far.

* Move travis CI to Python 3.7

* travis: use xenial dist

* fix newer linter warnings

Apparently bare excepts are literally Hitler, and we have some
new import sorting rules. Hooray!

* requirements: remove six

This is a dependency for sqlalchemy-utils, but we ourselves don't
depend on it directly because we've never been on Python 2 ever.

* Update requirements.txt
2019-08-10 18:39:53 -07:00
Nicolas F 16814d6eb7 Add trusted application functionality (#533)
* Add trusted application functionality

This lets users apply for trusted status, given certain minimum
requirements. Moderators can then review the applications, giving
a recommendation, and administrators can accept or reject them.

If an application is accepted or rejected, the user receives an
e-mail about it.

Markdown images are not rendered in applications to prevent browsers
from sending automatic requests to untrusted webservers.

Users who have had their application rejected cannot re-apply for a set
amount of days.

* minor fixes
2019-08-10 18:18:44 -07:00
nallown ff44d7a51c merge #481 add trusted and remake information to RSS feed 2019-08-10 23:32:38 +02:00
Nicolas F 90607d6993 main.css: fix stat styling on dark mode 2019-05-20 16:02:04 +02:00
Nicolas F 8614da6322 view.html: fix comment text copypasting in Firefox
Firefox would insert a lot of weird whitespace, because its
user-select would default to "all" and not "text", so it copypastes
the whitespace of otherwise completely invisible HTML elements.

I don't know why Firefox does this. It's stupid, and not what the
user expects. They're copying what they see, not what the DOM is.
2019-04-30 23:12:11 +02:00
Nicolas F 8f1925ce05 search_results: various fixes to the DOM
Avoid repeated style attributes, instead use nth-child CSS selectors
and classes where appropriate.

Also, get rid of the useless <div> around column headings. They don't
seem to do anything useful and are just more stuff for the browser's
layout engine to juggle around.

We can also use Jinja2 to remove some of the unnecessary whitespace
around table headings, which makes things like copy-pasting in Firefox
act less weird. This includes some other whitespace fixes too.
2019-04-30 23:11:40 +02:00
Nicolas F 38f8880966 Make blue info bubble dismissible, configurable (#526)
Infobubble text is now in a separate file, along with a timestamp
in the main file, so that changes to it don't result in merge
conflicts too often.

We also add some JS to make the bubble dismissible, keeping track
of the last timestamp that was dismissed in localstorage.

A timestamp of 0 disables the infobubble altogether.
2019-04-08 19:50:30 -07:00
Nicolas F 7de7147743 Query limit fixes (#555)
* Fix total torrent count for user listings

The total count would previously be bound by the maximum number
of pages we allow. Since we run the count query anyway, we can
just save this result and use it in the template.

* search: allow users to view all their uploads

Maximum page limitations shouldn't take effect when users are looking
at a listing of their own torrents.
2019-04-08 16:59:30 -07:00
TheAMM 885cccca40 Cache url_for calls with lru_cache
This commit adds a caching_url_for using functools.lru_cache (currently
with maxsize at 4096 entries) and replaces flask.url_for with it, as
there is no harm in doing so.

This greatly improves template generation speed, from ~115ms to ~75ms on
the front page (using the simple benchmark introduced in the previous
commit).
2019-04-08 19:28:26 +02:00
TheAMM a7558823d0 Further cleanup 2019-04-08 19:18:11 +02:00
Alex Ingram bd4eba9ab7 Change options in models.py based on AMM feedback. 2019-04-08 19:18:11 +02:00
Alex Ingram 75e7e942fb Fix some JOIN insanity with SQLAlchemy. 2019-04-08 19:18:11 +02:00
TheAMM b0c51e9fa0 Add very simple benchmarking support when debugging 2019-04-08 19:31:34 +03:00
Nicolas F db83989d5d extensions: limit pagination to max num of pages 2019-04-02 18:57:56 +02:00
Nicolas F 6608c99b90 search: allow limiting the maximum number of pages 2019-04-02 17:56:43 +02:00
Nicolas F 2782589cda config: add live.no to email blacklist 2019-03-30 16:33:08 +01:00
Nicolas F 6a63ee241a templates: cache torrent view filelist (#542)
* templates: cache torrent view filelist

Using flask-caching, we can add a 1 hour cache to the template
output of a filelist, varying it by the key "filelist" + the
hex infohash of a torrent.

Using a very big filelist as a test, I get a difference in page
load speeds of about a magnitude. (400ms -> 37 ms)

* templates: increase filelist cache to 24 hours
2019-03-08 06:29:51 -08:00
Nicolas F 585e1aeae7 tests: add tests for XML char replacement 2019-02-25 12:27:09 +01:00
TheAMM d7f9618fbf upload/edit: sanitize some string fields
This commit introduces a regex to replace illegal (expectedly unused)
characters from torrent display name, information link and description
upon upload or edit.

Fixes #541
2019-02-25 12:27:09 +01:00
Alex Ingram 1374375a16 Fix category icon display issue. (#547) 2019-02-24 15:07:49 +01:00
Nicolas F 51192342a6
view: fix template throwing exception on edge case (#549)
If a user has a comment under the edit time limit in a comment
locked torrent, but also are still affected by the new account CAPTCHA
cooldown, the template would throw an error as we tried to getattr on
a None object (namely, the comment_form).

To fix this, we also need to check around the edit form whether the
comment_form exists.
2019-02-24 15:03:38 +01:00
nyaadev 2e2e741d0f fix migration, add server_default for boolean 2019-01-24 12:31:48 +01:00
Alex Headley 18ebf134d5 feature: docker development environment (#544)
Add docker support

It's self-contained so should not interfere with anything else and can easily be thrown out if nobody wants it anymore.
2019-01-15 18:02:59 +01:00
Nicolas F a6d800b4ca templates: move icon_dir set outside of the loop
Someone put this inside the loop despite it essentially being
constant. Probably makes immeasurably little difference perf-wise,
but why not fix it anyway.
2019-01-12 20:15:15 +01:00
Anna-Maria Meriniemi 8365894268 config: add anonymous upload/registration limiting (#537) 2019-01-02 03:40:56 -08:00
AC e4780aa47d Replace leechers-paradise with desync (#536) 2018-12-13 18:05:26 -08:00
A nyaa developer 4fea839f51
fix user preferences from #515 (#532) 2018-12-06 15:47:08 +01:00
doge e04bbdfbf9 Replaced spaces with tabs 2018-11-20 13:45:31 +01:00
doge 92ca3a577f Generate database migration script 2018-11-20 13:45:31 +01:00
doge 3190394eea Comment Hiding
This commit adds the ability hide comments on torrent pages, and a user
preferences tab to allow registered users to make this the default
behavior.
2018-11-20 13:45:31 +01:00
Nicolas F 18bdf465f7 requirements: update dependencies for newer python
Apparently, when running things with Python 3.7, these three
dependencies ran into build issues with a renamed struct field.
Upgrading them seems to fix the issue, and hopefully keeps them
working with Python 3.6 as well.
2018-11-11 02:52:33 +01:00
Nalapl3 eabe13e20a Add amount of comments to RSS (#528)
Closes #496.
2018-10-27 17:17:13 +03:00
Nicolas F c967b831c6 backend: blacklist Windows reserved filenames (#527)
Windows has a few special filenames that it does not allow the
explorer.exe and command line to see, but can still be created by
applications. This is due to some jank DOS compatibility.

These filenames can be abused to troll Windows users, so we should
probably blacklist them.
2018-10-27 17:13:44 +03:00
TheAMM b2ddba994c Fix open redirect (#519)
The funny thing is that we don't even use this anywhere, and the
referrer is useless on forms. But hey, maybe someday.
2018-09-16 21:16:35 +03:00
Anna-Maria Meriniemi 8c892f09cc config/forms: allow blacklisting email servers by IP (#518)
Adds a new config entry (EMAIL_SERVER_BLACKLIST, tuple of IPv4 addresses
as strings) and an email validator for registering, which will query all
the MX records for the domain and reject the registration if any of the
A records for any of the MX records are found in the blacklist.
If the query fails, the blacklist is ignored; the email is accepted.
2018-08-20 11:29:30 -07:00
Nicolas F 86b7eb7ccd Revert "Make blue info bubble dismissible, configurable (#508)"
This reverts commit 62ec8a5236.
2018-08-03 04:34:19 +02:00
Nicolas F 62ec8a5236 Make blue info bubble dismissible, configurable (#508)
Infobubble text is now in a separate file, along with a timestamp,
so that changes to it don't result in merge conflicts too often.

We also add some JS to make the bubble dismissible, keeping track
of the last timestamp that was dismissed in localstorage.
2018-08-02 19:16:28 -07:00
TheAMM bf090c0fab torrents: clean up & optimize magnets further
Removes the specialized template ES magnet creator, since create_magnet()
can use both Torrents and ES objects. Search results will get the
properly escaped magnets, now.
Slightly optimizes the tracker adding and string joins.
RIP base32, wonder how many bad clients will break with sha1.
2018-07-21 22:06:58 +03:00
Tadeo Kondrak ef7ff1b9ce Add torrent counter on profile page (#409) (#513) 2018-07-17 14:52:54 +03:00
TheAMM 2bf96196cb torrents: Clean up magnet URI quotation
A cosmetic change.
Swapping quote_via to quote, we don't convert spaces to pluses, keeping
the name intact (which only mattered until peers send the metadata).
2018-07-15 04:40:07 +03:00