add examples and update README

README.md

@@ -1,5 +1,5 @@
 # 🤗 PET
-Parameter-Efficient Tuning. Intergrated with 🤗 Accelerate to scale seamlessly to large models using PyTorch FSDP. 
+Parameter-Efficient Tuning methods enable . Intergrated with 🤗 Accelerate to scale seamlessly to large models using PyTorch FSDP. 
 
 Supported methods:
 
@@ -38,19 +38,56 @@ For scaling to large models, you can leverage 🤗 Accelerate's PyTorch FSDP int
 PyTorch FSDP shards parameters, gradients and optimizer states across data parallel workers which enables
 large language models to fit on available hardware. 
 It also supports CPU offloading to further enable distributed training at scale. 
-The support for DeepSpeed ZeRO Stage-3 is currently in backlog.
 
 ```python
 from pet.utils.other import fsdp_auto_wrap_policy
 
 ...
 
-if accelerator.state.fsdp_plugin is not None:
+if os.environ.get("ACCELERATE_USE_FSDP", None) is not None:
     accelerator.state.fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(model)
 
 model = accelerator.prepare(model)
 ```
 
+Example of parameter efficient tuning with `mt0-xxl` base model using 🤗 Accelerate is provided in `~examples/pet_lora_seq2seq_accelerate_fsdp.py`. 
+1. First run `accelerate config --config_file fsdp_config.yaml` and answer the questionaire. 
+Below are the contents of the config file.
+```
+command_file: null
+commands: null
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: FSDP
+downcast_bf16: 'no'
+dynamo_backend: 'NO'
+fsdp_config:
+  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+  fsdp_backward_prefetch_policy: BACKWARD_PRE
+  fsdp_offload_params: true
+  fsdp_sharding_strategy: 1
+  fsdp_state_dict_type: FULL_STATE_DICT
+  fsdp_transformer_layer_cls_to_wrap: T5Block
+gpu_ids: null
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+megatron_lm_config: {}
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 2
+rdzv_backend: static
+same_network: true
+tpu_name: null
+tpu_zone: null
+use_cpu: false
+```
+2. run the below command to launch example script
+```
+accelerate launch --config_file fsdp_config.yaml examples/pet_lora_seq2seq_accelerate_fsdp.py
+```
+
 
 ## Models support matrix
 
@@ -85,5 +122,7 @@ model = accelerator.prepare(model)
 
 ## Caveats:
 1. Doesn't work currently with DeeSpeed ZeRO Stage-3. Extending support with DeeSpeed ZeRO Stage-3 is in backlog.
+2. When using `P_TUNING` or `PROMPT_TUNING` with `SEQ_2_SEQ` task, remember to remove the `num_virtual_token` virtual prompt predictions from the left side of the model outputs during evaluations. 
+