torrents.csv/README.md

134 lines
4.6 KiB
Markdown
Raw Normal View History

2018-10-04 23:33:33 +00:00
# Torrents.csv
2018-10-03 17:12:12 +00:00
2018-12-23 18:40:49 +00:00
<!-- Torrents.csv - An open source, collaborative repository of torrents, with a self-hostable web server. -->
2018-11-27 23:36:18 +00:00
[Demo Server](https://torrents-csv.ml)
2018-10-09 22:44:49 +00:00
`Torrents.csv` is a *collaborative* repository of torrents and their files, consisting of a searchable `torrents.csv`, and `torrent_files.csv`. With it you can search for torrents, or files within torrents. It aims to be a universal file system for popular data.
Its initially populated with a January 2017 backup of the pirate bay, and new torrents are periodically added from various torrents sites. It comes with a self-hostable [Torrents.csv webserver](https://torrents-csv.ml), a command line search, and a folder scanner to add torrents, and their files.
2018-10-03 17:12:12 +00:00
2018-12-02 19:00:16 +00:00
`Torrents.csv` will only store torrents with at least one seeder to keep the file small, will be periodically purged of non-seeded torrents, and sorted by infohash.
2018-10-11 21:59:30 +00:00
![img](https://i.imgur.com/yTFuwpv.png)
2018-10-11 22:11:26 +00:00
2018-11-27 23:36:18 +00:00
To request more torrents, or add your own, go [here](https://gitlab.com/dessalines/torrents.csv/issues).
2018-10-11 21:59:30 +00:00
Made with [Rust](https://www.rust-lang.org), [ripgrep](https://github.com/BurntSushi/ripgrep), [Actix](https://actix.rs/), [Inferno](https://www.infernojs.org), [Typescript](https://www.typescriptlang.org/).
2019-08-28 04:53:14 +00:00
2018-11-27 23:36:18 +00:00
## Webserver
2019-08-28 04:53:14 +00:00
2018-11-27 23:36:18 +00:00
`Torrents.csv` comes with a simple webserver. [Demo Server](https://torrents-csv.ml)
2019-08-28 04:58:01 +00:00
2019-03-18 19:31:32 +00:00
### Docker
2019-08-28 04:58:01 +00:00
2019-03-18 19:31:32 +00:00
```
git clone https://gitlab.com/dessalines/torrents.csv
2020-01-09 17:13:22 +00:00
cd torrents.csv
cd scripts && ./build_sqlite.sh -f && cd ..
2019-08-28 04:53:14 +00:00
cd docker/prod
docker-compose up -d
2019-03-18 19:31:32 +00:00
```
2019-08-28 04:53:14 +00:00
### Docker Development
2019-08-28 04:58:01 +00:00
```
2019-08-28 04:53:14 +00:00
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts && ./build_sqlite.sh && cd ..
cd docker/dev
docker-compose up -d
2019-08-28 04:58:01 +00:00
```
2019-08-28 04:53:14 +00:00
2019-03-18 19:31:32 +00:00
### Local
2019-08-28 04:53:14 +00:00
2019-03-18 19:31:32 +00:00
#### Requirements
2019-08-28 04:53:14 +00:00
2019-02-12 20:02:50 +00:00
- [Rust](https://www.rust-lang.org/)
- [Yarn](https://yarnpkg.com/en/)
- [SQLite3](https://www.sqlite.org/index.html)
2019-08-28 04:53:14 +00:00
2019-03-18 19:31:32 +00:00
#### Running
2019-08-28 04:53:14 +00:00
```
2018-10-24 22:33:14 +00:00
git clone https://gitlab.com/dessalines/torrents.csv
2018-11-27 23:36:18 +00:00
cd torrents.csv/scripts
./webserver.sh
2018-10-24 22:33:14 +00:00
```
2019-06-25 23:33:12 +00:00
and goto http://localhost:8902
2018-10-24 22:33:14 +00:00
2019-03-18 19:31:32 +00:00
If running on a different host, run `export TORRENTS_CSV_ENDPOINT=http://whatever.com` to change the hostname, or use a reverse proxy with nginx or apache2.
2018-10-04 20:23:57 +00:00
2019-02-13 00:33:27 +00:00
The torrent data is updated daily, and to do so, run, or place this in a crontab:
`cd scripts && ./git_update.sh`
2019-02-12 20:02:50 +00:00
This updates the repository, and rebuilds the sqlite cache necessary for searching.
2018-11-27 23:36:18 +00:00
## Command Line Searching
2018-10-11 21:59:30 +00:00
### Requirements
2018-11-27 23:36:18 +00:00
- [ripgrep](https://github.com/BurntSushi/ripgrep)
### Running
2018-11-27 23:36:18 +00:00
```
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv
./search.sh "bleh season 1"
bleh season 1 (1993-)
seeders: 33
size: 13GiB
link: magnet:?xt=urn:btih:INFO_HASH_HERE
```
## Uploading / Adding Torrents from a Directory
2019-08-28 04:53:14 +00:00
An *upload*, consists of making a pull request after running the `scan_torrents.sh` script, which adds torrents from a directory you choose to the `.csv` file, after checking that they aren't already there, and that they have seeders. It also adds their files to `torrent_files.csv`.
2018-10-05 06:48:51 +00:00
### Requirements
- [Torrent-Tracker-Health Dessalines branch](https://github.com/dessalines/torrent-tracker-health)
- `npm i -g dessalines/torrent-tracker-health`
2019-02-12 23:08:05 +00:00
- [jq command line JSON parser: Needs at least jq-1.6](https://stedolan.github.io/jq/)
2019-02-12 20:02:50 +00:00
- [NodeJS](https://nodejs.org/en/)
- [Gnu Parallel](https://www.gnu.org/software/parallel/)
### Running
2018-10-05 06:48:51 +00:00
[Click here](https://gitlab.com/dessalines/torrents.csv/forks/new) to fork this repo.
```sh
2018-10-04 23:33:33 +00:00
git clone https://gitlab.com/[MY_USER]/torrents.csv
cd torrents.csv/scripts
./scan_torrents.sh MY_TORRENTS_DIR # `MY_TORRENTS_DIR` is `~/.local/share/data/qBittorrent/BT_backup/` for qBittorrent on linux, but you can search for where your torrents are stored for your client.
2018-11-27 23:36:18 +00:00
git commit -am "Adding [MY_USER] torrents"
git push
```
2018-10-04 20:23:57 +00:00
2018-10-04 23:33:33 +00:00
Then [click here](https://gitlab.com/dessalines/torrents.csv/merge_requests/new) to do a pull/merge request to my branch.
2018-11-27 23:36:18 +00:00
## Web scraping torrents
`Torrents.csv` has a `Rust` repository for scraping new and top torrents from some torrent sites in the `new_torrents_fetcher` folder. It currently scrapes skytorrents, magnetdl, and leetx.
2018-11-27 23:36:18 +00:00
### Requirements
- Rust
2019-02-12 20:02:50 +00:00
- [Cloudflare Scrape](https://github.com/Anorov/cloudflare-scrape)
- `sudo pip install cfscrape`
2018-11-27 23:36:18 +00:00
### Running
```
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts
./update.sh SAVE_TORRENT_DIR
2018-11-27 23:36:18 +00:00
```
2019-02-13 00:39:31 +00:00
## API
A JSON output of search results is available at:
2019-06-25 23:33:12 +00:00
http://localhost:8902/service/search?q=[QUERY]&size=[NUMBER_OF_RESULTS]&offset=[PAGE]
2019-02-13 00:39:31 +00:00
2019-02-12 20:02:50 +00:00
## How the torrents.csv file looks
```sh
infohash;name;size_bytes;created_unix;seeders;leechers;completed;scraped_date
# torrents here...
```
2018-10-03 17:12:12 +00:00
## How the torrent_files.csv looks
```sh
infohash;index;path;size_bytes
```