Commit Graph

20 Commits

Author SHA1 Message Date
Anna-Maria Meriniemi 0cc25c3569
[ES] Improve search term preprocessing to include literal groups (#477)
* Extend ES term preprocessing for OR groups

Implements handling "foo"|"bar" literal OR groups in the Elasticsearch
term preprocessor. Groups can be negated with -, but don't mesh with
precedence (like plain literals).

This is a partial hack, the real solution would be to parse the entire
search terms ourselves, with AND and OR groups, negations etc. But
having that work neatly with the simple_query_string would be bit of a
hassle.

* Update help.html search tips

since search (quoting strings) has changed a bit.
2018-04-15 09:53:36 +03:00
Anna-Maria Meriniemi 0b78428abc [ES Change] Improve Elasticsearch term quoting (#473)
* Optimize Elasticsearch fullword field

Since the main display_name field ngrams words up to 15 characters,
anything to and under that will already be indexed - the fullword field
(which we have for words longer than 15 characters) needs to index only
words longer than that.

* Preprocess ES terms for better literal matching

This commit adds a new .exact subfield to display_name, which holds a
barely-filtered version of the original title we can do "literal"
matching against. This is not real substring matching, but quoting
terms now actually does something!

Implements a simple preprocessor for the search terms to extract quoted
parts from the search terms, optionally prefixed with - to negate them.
The preprocessor will create a query that'll join all three query-types:
the simple_query_string, must-phrases and must-not-phrases.
2018-04-13 17:06:25 -07:00
nyaadev 8f9400bb5f Revert "[Schema change] Torrents flags bitflag column to indexed columns (#471)"
This reverts commit 41a2a32f66.

Performs worse in some cases than what we had before.
2018-04-08 08:36:42 +02:00
A nyaa developer 41a2a32f66 [Schema change] Torrents flags bitflag column to indexed columns (#471)
* convert torrent table flags column from bitflag to independent indexed columns

* elasticsearch integration (untested)

* improve performance
2018-04-07 22:44:53 -07:00
nyaadev 4091cba837 fix mysql error and bad colors in css 2017-09-19 01:32:45 +02:00
TheAMM 07b45622f4 Get rid of index name hardcoding, read from database (and cache)
Index names are not necessarily named 'ix_table_column', so inspect the real schema for index names. Results are cached in memory.
(I have no clue how, but mine are prefixed idx_ instead of ix_!)
Clears out the "sort" variable naming as well.
2017-08-31 19:12:50 +03:00
nyaadev 294a12700c fix mysql being shit 2017-08-31 03:36:29 +02:00
Kfir Hadas 87dd95f1e0 Refactor into an app factory [2 of 2] (#322)
* Replace all `from nyaa import app` imports with `app = flask.current_app` (or `from flask import current_app as app` where possible)
* Add a separate config object for top-level and class statements as `nyaa.extensions.config`
Required because those codes don't have app context at the time of evaluation/execution.
* Remove `routes.py` file and register all blueprints in `nyaa/__init__.py`
* Refactor `nyaa/__init__.py` into an app factory
* Update tools
* Update tests (temporary, will be replaced)
2017-08-01 21:02:08 +03:00
Kfir Hadas 0181d6cb33 Prepare for app factory [1 of 2] (#315)
* Move db, assets, debug toolbar and fix_paginate into nyaa.extensions
* Change all `from nyaa import db` imports to `from nyaa.extensions import db`
* Move `nyaa.torrents.create_magnet_from_es_info` context processor into template-utils blueprint
* Fix tools (wrap in `with app.app_context():` where needed)
2017-07-30 20:35:16 +03:00
Kfir Hadas b992467dad Apply isort & flake8 (#312)
* Update isort settings
* Apply import sorting (isort) on all files in nyaa/
* Fixed Flake8 errors in nyaa/ (see PR for list)
* Add isort to lint.sh and requirements.txt
2017-07-28 20:01:19 +03:00
Kfir Hadas 0887dde6fc Move /user/<user_name> route into a blueprint
Move supporting functions and variables into other files
* nyaa.utils:
  - cached_function
  - chain_get
* nyaa.search:
  - DEFAULT_MAX_SEARCH_RESULT
  - DEFAULT_PER_PAGE
  - SERACH_PAGINATE_DISPLAY_MSG
  - _generate_query_string
2017-07-27 14:14:00 +03:00
TheAMM 2d0cf7cbb4 [ES Schema change] Multi-field search display_name to match words over ngram limit
This fixes searching for "Machiavellianism", 16 chars ("Machiavellianis", 15 chars, worked previously).
Does not (seem to!) break anything, but requires a re-indexing of ES.
2017-06-05 17:29:00 +03:00
Kfir Hadas 3165389d52 Fix lint check + update lint script (#224)
* Fix PEP8 E301 on nyaa/models.py

* Add utils/ to lint checker

* Run of lint.sh + manual fixes

As suggested https://github.com/nyaadevs/nyaa/pull/157#issuecomment-305051428

* Fix backwards tick in README

* Updated script

* Update Travis config
2017-06-01 06:40:33 -05:00
TheAMM 2b331c307e Optimize MySQL COUNT queries and pagination
Also leaves 'Torrent.trackers' as 'select' for the joining, since we don't need it on listings
2017-05-29 18:27:34 +03:00
TheAMM 16194a12a3 Fix typo in ES search sort keys 2017-05-27 21:37:26 +03:00
Kfir Hadas aad5a0c6ea Update style
Add support for sorting comments (in advance),
add th for sorting by comments_count, add title to the counter box and fix it being under the torrent's name,
2017-05-26 18:25:56 +03:00
Kfir Hadas 2c9ed4cb94 PEP8 search.py (E501) 2017-05-16 12:52:48 +03:00
Kfir Hadas 4e9409fb30 AutoPEP8 (after elasticsearch merge) 2017-05-16 12:47:06 +03:00
aldacron 2005174358 finished up rss, changed rss behavior to include pre-defined trackers only, also cleaned up debug statements 2017-05-16 00:46:25 -07:00
aldacron 899aa01473 hooked up ES... 90% done, need to figure out how to generate magnet URIs 2017-05-15 23:51:58 -07:00