Some torrent delivery optimizations (#46)

* Disable IPv6 which doesn't work in Colab

* More tweaks

* Yeah max-connection-per-server doesn't work above 16

* Cosmetic

* Set a bigger number of peers

Apparently -x and -s do not affect this parameter.

* Enable local peer discovery

* Revert local peer discovery and IPv6

* Reorder things around.

* Cosmetic.

* Increase peer number further to 500

Increasing popularity doesn't do any good.
This commit is contained in:
Aleksej Komarov
2019-12-08 08:14:46 -07:00
committed by Nick Walton
parent 02cd136a92
commit 83c6a4e892
+19 -5
View File
@@ -11,15 +11,29 @@ MODEL_TORRENT_BASENAME="$(basename "${MODEL_TORRENT_URL}")"
if [[ -d "${MODELS_DIRECTORY}/${MODEL_VERSION}" ]]; then
echo "AIDungeon2 is already installed"
else
echo "Downloading AIDungeon2 Model... (this may take a few minutes)"
echo "Installing dependencies"
pip install -r requirements.txt > /dev/null
apt-get install aria2 unzip > /dev/null
echo "Downloading AIDungeon2 Model... (this may take a very, very long time)"
mkdir -p "${MODELS_DIRECTORY}"
cd "${MODELS_DIRECTORY}"
mkdir "${MODEL_VERSION}"
apt-get install aria2 unzip > /dev/null
wget "${MODEL_TORRENT_URL}"
unzip "${MODEL_TORRENT_BASENAME}"
aria2c -x 16 -s 32 --seed-time=0 "${MODEL_TORRENT_BASENAME%.*}"
echo -e "\n\n==========================================="
echo "We are now starting to download the model."
echo "It will take a while to get up to speed."
echo "After download completes, we will seed this for 1 minute to ensure high availability."
echo "DHT errors are normal."
echo -e "===========================================\n"
aria2c \
--max-connection-per-server 16 \
--split 64 \
--bt-max-peers 500 \
--seed-time=1 \
--summary-interval=15 \
--disable-ipv6 \
"${MODEL_TORRENT_BASENAME%.*}"
echo "Download Complete!"
cd "${BASE_DIR}"
pip install -r requirements.txt > /dev/null
fi