Commit Graph

16 Commits

Author SHA1 Message Date
queue 4fcef92b94
elasticsearch 7.x compatability (#576)
* es_mapping: update turning off dynamic mappings

they changed it in 6.x

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
https://github.com/elastic/elasticsearch/pull/25734

* es_mapping: remove _all field

deprecated in 6.0 anyway

* es_mapping.yml: fix deprecated mapping type

https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html#_schedule_for_removal_of_mapping_types

it gives a really unhelpful error otherwise, oof.

* es: fix remaining 7.xisms

the enabled: false apparently only applies to
"object" fields now, need index: false

and the _type got removed everywhere. Seems to work now.

* Fix weird offset error with word_delimiter_graph

yet another es7-ism i guess

* Fix warning and some app stuff for ES 7.x

Co-authored-by: Arylide <Arylide@users.noreply.github.com>
2020-07-12 00:10:47 -07:00
Anna-Maria Meriniemi 5c8b119611 config: Add Elasticsearch hosts (#492) 2018-07-09 22:26:23 -07:00
TheAMM 81806d7bc9 Pad info_hash in ElasticSearch sync scripts
python-mysql-replication (or PyMySQL) would return less than 20 bytes
for info-hashes that had null bytes near the end, leaving incomplete
hashes in the ES index. Without delving too deep into the real issue
(be it lack of understanding MySQL storing binary data or a bug in
the libraries), thankfully we can just pad the fixed-size info-hashes
to be 20 bytes.

Padding in import_to_es.py may be erring on the side of caution, but
safe is established to be better than sorry.

(SQLAlchemy is unaffected by this bug)

Fixes #456
2018-02-25 15:12:35 +02:00
nyaadev 1bc21bc599 fix broken import_to_es.py 2017-09-17 02:54:09 +02:00
Kfir Hadas 87dd95f1e0 Refactor into an app factory [2 of 2] (#322)
* Replace all `from nyaa import app` imports with `app = flask.current_app` (or `from flask import current_app as app` where possible)
* Add a separate config object for top-level and class statements as `nyaa.extensions.config`
Required because those codes don't have app context at the time of evaluation/execution.
* Remove `routes.py` file and register all blueprints in `nyaa/__init__.py`
* Refactor `nyaa/__init__.py` into an app factory
* Update tools
* Update tests (temporary, will be replaced)
2017-08-01 21:02:08 +03:00
Kfir Hadas 0181d6cb33 Prepare for app factory [1 of 2] (#315)
* Move db, assets, debug toolbar and fix_paginate into nyaa.extensions
* Change all `from nyaa import db` imports to `from nyaa.extensions import db`
* Move `nyaa.torrents.create_magnet_from_es_info` context processor into template-utils blueprint
* Fix tools (wrap in `with app.app_context():` where needed)
2017-07-30 20:35:16 +03:00
TheAMM 0cbed3870e Update import_to_es.py to index both torrent flavors, rename sync_es config
import_to_es.py will now also SHOW MASTER STATUS now
Changed progressbar dependency
2017-05-28 02:12:48 +03:00
TheAMM 9cd6c506ae Update ElasticSeach index and scripts for comment_count 2017-05-26 16:12:47 +03:00
nyaadev 152e547ac5 Add flask-Migrate + alembic for automated database migrations.
Update some dependencies to their latest version.
Make executable scripts executable (chmod +x).
2017-05-21 17:47:16 +02:00
aldacron f27cf17478 added timeout to import and sync es 2017-05-16 23:15:48 -07:00
queue 4a777376a2 import_to_es: fix put_settings not being strings
annoying.
2017-05-16 23:06:07 -06:00
queue c4dbbdae7b import_to_es: munge refresh_interval for speed 2017-05-16 23:00:58 -06:00
aldacron 899aa01473 hooked up ES... 90% done, need to figure out how to generate magnet URIs 2017-05-15 23:51:58 -07:00
aldacron c2c547e786 some more elasticsearch work, including index mapping and analyzer 2017-05-15 11:14:01 -07:00
queue 3cbe2e4221 WIP hack in es as the provider for search results
real sketch. lots of stuff is still broken. But! you can
make elasticsearch q= style queries and it shows up properly.
only first page works; need to adapt pager to elasticsearch's "total-hits" thing.
2017-05-14 02:01:26 -06:00
queue 17217d9427 WIP es stuff 2017-05-14 00:48:17 -06:00