[sgd] Extend distributed pytorch functionality (#5675)

* raysgd

* apply fn

* double quotes

* removed duplicate TimerStat

* removed duplicate find_free_port

* imports in pytorch_trainer

* init doc

* ray.experimental

* remove resize example

* resnet example

* cifar

* Fix up after kwargs

* data_dir and dataloader_workers args

* formatting

* loss

* init

* update code

* lint

* smoketest

* better_configs

* fix

* fix

* fix

* train_loader

* fixdocs

* ok

* ok

* fix

* fix_update

* fix

* fix

* done

* fix

* fix

* fix

* small

* lint

* fix

* fix

* fix_test

* fix

* validate

* fix

* fi

doc/source/distributed_training.rst

@@ -12,16 +12,22 @@ Wrap your training with this:
 .. code-block:: python
 
     ray.init(args.address)
+
     trainer1 = PyTorchTrainer(
         model_creator,
         data_creator,
         optimizer_creator,
+        loss_creator,
         num_replicas=<NUM_GPUS_YOU_HAVE> * <NUM_NODES>,
         use_gpu=True,
         batch_size=512,
-        backend="gloo")
+        backend="nccl")
+
+    stats = trainer1.train()
+    print(stats)
+    trainer1.shutdown()
+    print("success!")
 
-    trainer1.train()
 
 
 Then, start a Ray cluster `via autoscaler <autoscaling.html>`_ or `manually <using-ray-on-a-cluster.html>`_.