6493de4a5b | ||
---|---|---|
new_torrents_fetcher | ||
scripts | ||
server | ||
.dockerignore | ||
.gitattributes | ||
.gitignore | ||
Dockerfile | ||
LICENSE | ||
README.md | ||
docker-compose.yml | ||
search.sh |
README.md
Torrents.csv
Torrents.csv
is a collaborative repository of torrents and their files, consisting of a searchable torrents.csv
, and torrent_files.json
. With it you can search for torrents, or files within torrents. It aims to be a universal file system for popular data.
Its initially populated with a January 2017 backup of the pirate bay, and new torrents are periodically added from various torrents sites. It comes with a self-hostable Torrents.csv webserver, a command line search, and a folder scanner to add torrents, and their files.
Torrents.csv
will only store torrents with at least one seeder to keep the file small, will be periodically purged of non-seeded torrents, and sorted by infohash.
To request more torrents, or add your own, go here.
Made with Rust, ripgrep, Actix, Inferno, Typescript.
Webserver
Torrents.csv
comes with a simple webserver. Demo Server
Docker
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv
docker-compose up
Local
Requirements
Running
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts
./webserver.sh
and goto http://localhost:8080
If running on a different host, run export TORRENTS_CSV_ENDPOINT=http://whatever.com
to change the hostname, or use a reverse proxy with nginx or apache2.
The torrent data is updated daily, and to do so, run, or place this in a crontab:
cd scripts && ./git_update.sh
This updates the repository, and rebuilds the sqlite cache necessary for searching.
Command Line Searching
Requirements
Running
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv
./search.sh "bleh season 1"
bleh season 1 (1993-)
seeders: 33
size: 13GiB
link: magnet:?xt=urn:btih:INFO_HASH_HERE
Uploading / Adding Torrents from a Directory
An upload, consists of making a pull request after running the scan_torrents.sh
script, which adds torrents from a directory you choose to the .csv
file, after checking that they aren't already there, and that they have seeders. It also adds their files to torrent_files.json
.
Requirements
- Torrent-Tracker-Health Dessalines branch
npm i -g dessalines/torrent-tracker-health
- jq command line JSON parser: Needs at least jq-1.6
- NodeJS
- Gnu Parallel
Running
Click here to fork this repo.
git clone https://gitlab.com/[MY_USER]/torrents.csv
cd torrents.csv/scripts
./scan_torrents.sh MY_TORRENTS_DIR # `MY_TORRENTS_DIR` is `~/.local/share/data/qBittorrent/BT_backup/` for qBittorrent on linux, but you can search for where your torrents are stored for your client.
git commit -am "Adding [MY_USER] torrents"
git push
Then click here to do a pull/merge request to my branch.
Web scraping torrents
Torrents.csv
has a Rust
repository for scraping new and top torrents from some torrent sites in the new_torrents_fetcher
folder. It currently scrapes skytorrents, magnetdl, and leetx.
Requirements
- Rust
- Cloudflare Scrape
sudo pip install cfscrape
Running
git clone https://gitlab.com/dessalines/torrents.csv
cd torrents.csv/scripts
./update.sh SAVE_TORRENT_DIR
API
A JSON output of search results is available at:
http://localhost:8080/service/search?q=[QUERY]&size=[NUMBER_OF_RESULTS]&offset=[PAGE]
How the torrents.csv file looks
infohash;name;size_bytes;created_unix;seeders;leechers;completed;scraped_date
# torrents here...
How the torrent_files.json looks
{
"012ae083ec82bf911f4fe503b9f6df1effaad9ac": [
{
"i": 0, // the index
"p": "File 1", // the path
"l": 88546036A // the size in bytes
},
...
]
}