Go to file
Rob Watson b727949dcd tidy up 2020-07-04 09:32:36 +02:00
docker Upping version. 2020-06-21 16:20:20 -04:00
new_torrents_fetcher Adding /bin/bash to script. 2020-01-08 15:56:46 -05:00
qBittorrent Adding qBittorrent search plugin. 2019-09-23 17:58:29 -07:00
scripts correcting build_sqlite. 2020-01-26 21:58:51 -05:00
server tidy up 2020-07-04 09:32:36 +02:00
.dockerignore Docker updates. 2019-04-10 15:48:23 -07:00
.gitattributes Data Updates 2019-03-09 11:13:23 -08:00
.gitignore Adding an exit to scan torrents. 2019-02-19 18:04:09 -08:00
LICENSE Script mostly working. 2018-10-04 13:23:57 -07:00
README.md Updating deps, adding /new 2020-06-06 18:35:34 -04:00
search.sh Import of TPB Jan 2017 backup. Updating search and add scripts. Adding prune.sh 2018-10-07 22:43:12 -07:00
torrent_files.csv Data updates. 2020-05-30 23:45:44 -04:00
torrents.csv Data updates 2020-06-05 21:32:34 -04:00

README.md

Torrents.csv

Demo Server

Torrents.csv is a collaborative repository of torrents and their files, consisting of a searchable torrents.csv, and torrent_files.csv. With it you can search for torrents, or files within torrents. It aims to be a universal file system for popular data.

Its initially populated with a January 2017 backup of the pirate bay, and new torrents are periodically added from various torrents sites. It comes with a self-hostable Torrents.csv webserver, a command line search, and a folder scanner to add torrents, and their files.

Torrents.csv will only store torrents with at least one seeder to keep the file small, will be periodically purged of non-seeded torrents, and sorted by infohash.

img

To request more torrents, or add your own, go here.

Made with Rust, ripgrep, Actix, Inferno, Typescript.

Webserver

Torrents.csv comes with a simple webserver. Demo Server

Docker

git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv
cd scripts && ./build_sqlite.sh -f && cd ..
cd docker/prod
docker-compose up -d

Docker Development

git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts && ./build_sqlite.sh && cd ..
cd docker/dev
docker-compose up -d

Local

Requirements

Running

git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts
./webserver.sh

and goto http://localhost:8902

If running on a different host, run export TORRENTS_CSV_ENDPOINT=http://whatever.com to change the hostname, or use a reverse proxy with nginx or apache2.

The torrent data is updated daily, and to do so, run, or place this in a crontab:

cd scripts && ./git_update.sh

This updates the repository, and rebuilds the sqlite cache necessary for searching.

To re-build the frontend assets, use cd server/ui && yarn build. There is no need to restart the ./webserver.sh script.

Command Line Searching

Requirements

Running

git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv
./search.sh "bleh season 1"
bleh season 1 (1993-)
	seeders: 33
	size: 13GiB
	link: magnet:?xt=urn:btih:INFO_HASH_HERE

Uploading / Adding Torrents from a Directory

An upload, consists of making a pull request after running the scan_torrents.sh script, which adds torrents from a directory you choose to the .csv file, after checking that they aren't already there, and that they have seeders. It also adds their files to torrent_files.csv.

Requirements

Running

Click here to fork this repo.

git clone https://gitlab.com/[MY_USER]/torrents.csv
cd torrents.csv/scripts
./scan_torrents.sh MY_TORRENTS_DIR # `MY_TORRENTS_DIR` is `~/.local/share/data/qBittorrent/BT_backup/` for qBittorrent on linux, but you can search for where your torrents are stored for your client.
git commit -am "Adding [MY_USER] torrents"
git push

Then click here to do a pull/merge request to my branch.

Web scraping torrents

Torrents.csv has a Rust repository for scraping new and top torrents from some torrent sites in the new_torrents_fetcher folder. It currently scrapes skytorrents, magnetdl, and leetx.

Requirements

Running

git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts
./update.sh SAVE_TORRENT_DIR

API

A JSON output of search results is available at:

http://localhost:8902/service/search?q=[QUERY]&size=[NUMBER_OF_RESULTS]&offset=[PAGE]&type=[torrent | file]

new torrents are at:

http://localhost:8902/service/new?size=[NUMBER_OF_RESULTS]&offset=[PAGE]&type=[torrent | file]

How the torrents.csv file looks

infohash;name;size_bytes;created_unix;seeders;leechers;completed;scraped_date
# torrents here...

How the torrent_files.csv looks

infohash;index;path;size_bytes