Commit Graph

14 Commits

Author SHA1 Message Date
TheAMM 81806d7bc9 Pad info_hash in ElasticSearch sync scripts
python-mysql-replication (or PyMySQL) would return less than 20 bytes
for info-hashes that had null bytes near the end, leaving incomplete
hashes in the ES index. Without delving too deep into the real issue
(be it lack of understanding MySQL storing binary data or a bug in
the libraries), thankfully we can just pad the fixed-size info-hashes
to be 20 bytes.

Padding in import_to_es.py may be erring on the side of caution, but
safe is established to be better than sorry.

(SQLAlchemy is unaffected by this bug)

Fixes #456
2018-02-25 15:12:35 +02:00
nyaadev 1bc21bc599 fix broken import_to_es.py 2017-09-17 02:54:09 +02:00
Kfir Hadas 87dd95f1e0 Refactor into an app factory [2 of 2] (#322)
* Replace all `from nyaa import app` imports with `app = flask.current_app` (or `from flask import current_app as app` where possible)
* Add a separate config object for top-level and class statements as `nyaa.extensions.config`
Required because those codes don't have app context at the time of evaluation/execution.
* Remove `routes.py` file and register all blueprints in `nyaa/__init__.py`
* Refactor `nyaa/__init__.py` into an app factory
* Update tools
* Update tests (temporary, will be replaced)
2017-08-01 21:02:08 +03:00
Kfir Hadas 0181d6cb33 Prepare for app factory [1 of 2] (#315)
* Move db, assets, debug toolbar and fix_paginate into nyaa.extensions
* Change all `from nyaa import db` imports to `from nyaa.extensions import db`
* Move `nyaa.torrents.create_magnet_from_es_info` context processor into template-utils blueprint
* Fix tools (wrap in `with app.app_context():` where needed)
2017-07-30 20:35:16 +03:00
TheAMM 0cbed3870e Update import_to_es.py to index both torrent flavors, rename sync_es config
import_to_es.py will now also SHOW MASTER STATUS now
Changed progressbar dependency
2017-05-28 02:12:48 +03:00
TheAMM 9cd6c506ae Update ElasticSeach index and scripts for comment_count 2017-05-26 16:12:47 +03:00
nyaadev 152e547ac5 Add flask-Migrate + alembic for automated database migrations.
Update some dependencies to their latest version.
Make executable scripts executable (chmod +x).
2017-05-21 17:47:16 +02:00
aldacron f27cf17478 added timeout to import and sync es 2017-05-16 23:15:48 -07:00
queue 4a777376a2 import_to_es: fix put_settings not being strings
annoying.
2017-05-16 23:06:07 -06:00
queue c4dbbdae7b import_to_es: munge refresh_interval for speed 2017-05-16 23:00:58 -06:00
aldacron 899aa01473 hooked up ES... 90% done, need to figure out how to generate magnet URIs 2017-05-15 23:51:58 -07:00
aldacron c2c547e786 some more elasticsearch work, including index mapping and analyzer 2017-05-15 11:14:01 -07:00
queue 3cbe2e4221 WIP hack in es as the provider for search results
real sketch. lots of stuff is still broken. But! you can
make elasticsearch q= style queries and it shows up properly.
only first page works; need to adapt pager to elasticsearch's "total-hits" thing.
2017-05-14 02:01:26 -06:00
queue 17217d9427 WIP es stuff 2017-05-14 00:48:17 -06:00