mirror of
https://github.com/wassname/pytorch-lightning.git
synced 2026-09-20 13:10:42 +08:00
William Falcon
396047ffa0
* added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * added simple cluster template * sets correct backend for possible combinations of gpu inputs * sets correct backend for possible combinations of gpu inputs * simple slurm example * simple slurm example * simple slurm example
Multi-node examples
Use these templates for multi-node training
Simplest example.
- Modify this script with your CoolModel file.
- Update and submit this bash script
squeue minimal_multi_node_demo_script.sh
Grid search on a cluster
Option 1: Run on cluster using your own SLURM script
The trainer and model will work on a cluster if you configure your SLURM script correctly.
- Update this demo slurm script.
- Submit the script
$ squeue demo_script.sh
Most people have some way they automatically generate their own scripts.
To run a grid search this way, you'd need a way to automatically generate scripts using all the combinations of
hyperparameters to search over.
Option 2: Use test-tube for SLURM script
With test tube we can automatically generate slurm scripts for different hyperparameter options.
To run this demo:
source activate YourCondaEnv
python multi_node_cluster_auto_slurm.py --email your@email.com --gpu_partition your_partition --conda_env YourCondaEnv
That will submit 6 jobs. Each job will have a specific combination of hyperparams. Each job will also run on 2 nodes where each node has 8 gpus.