wassname/jsteer
1
0
Fork 0
mirror of https://github.com/wassname/jsteer.git synced 2026-09-09 11:25:03 +08:00
Code Issues Packages Projects Releases Wiki Activity

U4 step3 guard: back off when GPU/RAM in use, yield to the live user

Browse Source 
Discovered the user is actively developing jsteer (commit 80353ed 90s ago,
live 16GB VRAM kernel from their demo work) -- they are NOT afk. The blind
requeue guard would compete with their interactive work and risk oomd-killing
THEIR process. Now the guard only launches the fit when GPU free >=13GB and
host avail >=20GB, so it fills genuinely-idle windows (overnight) and never
fights the human for their own machine. Killed the competing fit (555) to
yield the GPU now; checkpoint preserved at n_done=69.

Co-Authored-By: Claudypoo <288921227+claudypoo@users.noreply.github.com>
This commit is contained in:
wassname
2026-07-10 14:53:35 +08:00
co-authored by Claudypoo
parent 17f5310660
commit 57a0276ae6
1 changed files with 22 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/u4_step3_guard.sh
+22 -8
View File
@@ -7,25 +7,39 @@
# u4_step3_retry.sh dies with it (observed: task 554 Killed whole, no retry line).
#
# This guard runs DETACHED in its own session/cgroup, uses ~0 RAM (so oomd never
# targets it), and keeps the resumable fit queued until it completes. jlens
# checkpoints every prompt (n_done monotonic: 36->45->64->69 across kills), so
# each oomd kill only costs a model reload. It grinds slowly during the user's
# active hours and finishes clean overnight once their kernel idles.
# targets it), and launches the resumable fit ONLY into genuinely-idle GPU/RAM
# windows (see backoff below) so it never competes with the user's live work.
# jlens checkpoints every prompt (n_done monotonic: 36->45->64->69 across kills),
# so any kill only costs a model reload. Completes clean overnight once the
# user's kernel idles; does nothing (polite) while they are active.
#
# Stops itself the instant artifacts/u4_loopclose.txt exists (success). To stop
# early: pkill -f u4_step3_guard.sh
set -u
cd /media/wassname/SGIronWolf/projects5/2026/jspace/jsteer
LABEL='U4 step3 guard-requeue'
# POLITE backoff: the user is developing jsteer live (their VS Code GPU kernel is
# the oomd competitor). Only launch the fit when the GPU/host is genuinely idle,
# so we NEVER compete with the user's interactive work -- we just fill the empty
# windows (overnight, breaks). This wastes no idle GPU time yet never fights the
# human for their own machine. Thresholds sized for the fit's ~10.5GB footprint.
GPU_FREE_MIN_MIB=13000 # need >=13GB free (i.e. others using <~11GB) to start
RAM_AVAIL_MIN_MIB=20000 # need >=20GB host available (oomd triggers on host too)
while [ ! -f artifacts/u4_loopclose.txt ]; do
active=$(pueue status --json | jq -r --arg l "$LABEL" \
'.tasks[] | select((.label // "") | contains($l)) | .status | if type=="object" then keys[0] else . end' \
| grep -Ec 'Running|Queued|Stashed|Paused')
if [ "$active" -eq 0 ]; then
pueue add \
-l "why: $LABEL -- oomd-resilient external requeue of the 4B fit; resolve: resume checkpoint to n_done=400, writes u4_loopclose.txt on success" \
-w "$PWD" -o 0 -- bash scripts/u4_step3_retry.sh
echo "$(date '+%F %H:%M:%S') requeued fit (was not active)"
gpu_free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | head -1)
ram_avail=$(free -m | awk '/^Mem:/{print $7}')
if [ "$gpu_free" -ge "$GPU_FREE_MIN_MIB" ] && [ "$ram_avail" -ge "$RAM_AVAIL_MIN_MIB" ]; then
pueue add \
-l "why: $LABEL -- polite idle-window fit of the 4B Jacobian; resolve: resume checkpoint to n_done=400, writes u4_loopclose.txt on success" \
-w "$PWD" -o 0 -- bash scripts/u4_step3_retry.sh
echo "$(date '+%F %H:%M:%S') GPU idle (free=${gpu_free}MiB, ram=${ram_avail}MiB) -- launched fit"
else
echo "$(date '+%F %H:%M:%S') user active (gpu_free=${gpu_free}MiB, ram_avail=${ram_avail}MiB) -- backing off"
fi
fi
sleep 300
done