From 1bc55e182dafdcbdd123f2f7b42939bc83efdf5d Mon Sep 17 00:00:00 2001 From: Eric Liang Date: Mon, 1 Jan 2018 13:02:05 -0800 Subject: [PATCH] Update the pip wheel in example.yaml and add docs (#1381) --- doc/source/autoscaler-status.png | Bin 0 -> 20475 bytes doc/source/autoscaling.rst | 80 +++++++++++++++----- doc/source/using-ray-on-a-cluster.rst | 4 + doc/source/using-ray-on-a-large-cluster.rst | 4 + python/ray/autoscaler/aws/example.yaml | 5 +- 5 files changed, 73 insertions(+), 20 deletions(-) create mode 100644 doc/source/autoscaler-status.png diff --git a/doc/source/autoscaler-status.png b/doc/source/autoscaler-status.png new file mode 100644 index 0000000000000000000000000000000000000000..5ee87decd6ab53204c7a96f8c58e8a1ee75ff749 GIT binary patch literal 20475 zcmb@ubyOTd(C9n3TW}}AU4uJ8mf-I065O32K?1?uEqHKucXti$1b29wd~)u0-h1c% zaqsLovpqB2wKY9G-POOU-Y^9@Nfbl^L;wI#KvJKS005E>oQ{Nt1n<|rVGDx4pv^^O zMF5~G8tK^(27F9rETtq10G?C;;2#74kKj}O2LRy00suz_0Kl6J0J!$)&5Hcs3$R9? zB|iage?Qr6MRDLW2zC+@P5=Nw3{JlQMsVyC-~^mANLCzf3jzZ82&*~I1Zf-8Dez^( z1Qkv|uHYUNVPKaBW@IZL`F{~!>9qL;nc_*B~~_8 z?OlFABUA`x!UQP@-R3^0F89{A>a4~3I#-gE9&XtQLIA8f%)mL zw!~itwsSJu=||E^l~}EYz|VuiP)Vk_?*P7-|l+490?j- zTAG`iTU+zF*y?jK<1beuL`S#3*bGD$f9S{%@VK`4YW4%zJ+v@J{ zaJh>Y!l+eiQP@+V*Crji)3D)te0q9%a>BnqQ&pDT*FZ`WLv^0%12>3H6aS-i>4!t6 z>HZ4%-eKY3)GKtFOon2`BXGJ#*^9wnjGEO!BfF@msCsRlE`qocu#LJ@pFA5s^D^fn zVo6J_s|=3T-#Q&m7nqJD%B~`hBr>)*AHEw^y-(xdj1^M&AiFN0CR`$%6SO01=~Z8l zI+Db!OcS3rY>9=qd$~K3nVBh6%i;dpZ-(6}#Q)u|r%5}gH2`3enqgmWD4Sqj-i*kZ zJ|s5Z*&bC#wT5U+i@40xMg&+c4fp5=U?HocpKr~tSmBra9M$I!S>M#7PqruQpTAxD z(H9hX)lwk=3Kpw%4nu^QS1wTgQ?F=3M^`?#O$lGvGvACB-h7Qbjyf{6JR7VgkER9F zA`r7$hB#w!1qI9V1u)0%8Zl2Po`&^@Y)QAz1Y>b6szY*65XVAs@AbnFChnVGD2zAP ze~~@Rt*rE}C7%0?ulxSoOyS!SxiY^%DVQkMX>lD}T3UL9qoyu;kTN$nx4y6)N1AYm zJSAmZOPFr|IHZHz>G`}vLf*^&(PO2;=%u51Z)e}y?P`rm7W#_g+sjXU6!(j~7rQwN z+H`wG$ORXfHNAyt-xJx9>aQKYo4z{}3#b`O!@DOQ^D2+tyxEz`MlkbPl`jOM$nEV< z+s~l#KPJ>xOVlOep2@dh-nq8LTiE?9^_xZFkDiJ+%4A<7B|cO1O?baP-a*Pf>0>Ao zF5gHKzz55c>)BCd9@&i$jiiGpe6hvQblg1N#8tB9_M)vHLlWxD;3DzQ=J;f{TlnHDmsd& zf+vp{%HFgR`^9X9y*4xbRGa`fh@r4U9eh<@rLHt*p^iAJ&4<4M26 z+U*_~cvW$du&xFs7p=GJ?Yvw?%%(Z@v2BnhX9rcOvK^p1JU+(k!`ffO4U>O7EqRaR zUn<{;w9?QfY(254D*yneO+x~TTVGK}_a$!C&&Iz5xUA^mm6jfoR5w=lG_<7>9Y6r& z(uXm|-vz{QvfVGbxjL!?&LM>L?(-k*Wn0V!Avxq`_tpp@1ZylOetA93|GU~$(c+&?}{-Wfe_E;nyT zdFm`s-s`(}=(59dWNvb`yz}_NbxzR`bafK8lX&bY)H;d?RIG9bpjlX#@P!olMko+_ zl@^+RqaPmk^e7jfQF3lHhW4+<17MehTz!5gj9Kdp)P9d`STAb*GrUKwGr;jZffP*+ z(7D%lP|QP|KN}ue+q1J0x>S>WyVupk!{*si`v`&nK2JPZZQR@Dnv)Pc9Bk&eqELFP ze+lU{&u)1%7<;%dwO*)!s3pBYmX`sFZPhjX>p9cKA=H_pqN6L5j(6%3`zY@fLkQ3s zkmXt)Hznu3QL#?an5D@UVe?=|f?0kH~eF7wuR;)jC9)x<*24Gp~? z*QnO_R#$b8vs<*hcjzHmvZ%QMWApW4pO(v)=$1zbi-3tTq)i5wD~uD*FchHHrE=%Y zclyP9OnYG5q753qsij-D&OD0f*<$j0QmZi|3b4p*x)aVFaXM2*Sj%yLkrh>ocHet8 zJT|u2%bHPK@oZs_s{K5N(a?I!?tv@;;UCa{CoA6(TgTHmmmfp=rHO@A#F?09Ha9ou ze0Y4wlc#f9*R8Q4tNu1Y)|v*0|Cm;-?-Ou1=xtC}Ror$~`Y}!4$24Ohxqeqlc7waK zPuUK%Barx!`E zmf1XlR>^rhBCS86&eC^5Ft(22#6FpkMo<<)N>PAJN+ydIx140n+F02#=R5uf17#h%L6*Aaku6lgHMI`Mz^9CHgwC#X(!j$>|EECunbPk2D`t`R8KJmd zQF61MYr~sf9jo)g?-rjFP=4!7K&YgicLHijeGDB5&fhdX!q(RPZfBCia_zQ$2U3Gv z=^GkSd&(EcAlijmPTgLkn$|;cA$T0HMDdsJ9zn2v2$-;4ZViDfWG|XaJa0eXg!r|s zhj_RDCR*#k2m??VbYF|-!FC=8H@XnX8Hf_W^*0l`Y-85ypWt*5o$Y_Tw){L6liRxS z1^0Bg@{p2>CbNA+2g1KPa_r*5S(6~)qVT?4-S~y)b>eXb2=VM7bTh5|lMDC{u#v1p z7%7kD!&22b_dy!qAE;)LI=n31ic!-$<+bhq>Z`Y1`NGoscki*EwiZx6oPQst_t-b# zd+U6nMc{u}8i7Mi9SaR8cN{(cm>Ir*p^gS|{-UilrRS#su%fZwLS>yCECg3lmK=sA z9-hj-Yo9~0%nR9jY^}&^)dGT5LsJ0|znMEy+pN`z;A##8d5s~%{Q^7v`X(~J>q|n} zeqPAV(;eih_vjuPvg}?Pzy|u6^x7p|HF#<_(GUx~x&k?Q`l9640~Cn>+XzSZ?p$m2 z?+~`9H4Ne9E7$VhuI?^=cl?2bGmx3{acJ$>jYOcaey6ut^V0opVM_REsz~f=`aw2$ zbn8LJ=u5x~+Y>}K!|&eq+QPQlHl_Cdjn{IAxaJG{^12Ym9*waC32kH`LYEf6Vm{8Y z#PVk)qaF+kU$AGmUbRyv%2f8gmVTy+uSY}4CVT_wW6U`_G5_IyE`rucsPL%z1Xs*>25bMK7s;ANT;Kb8j!b*T7?N3}k=YbB;vSx}(t5~FNG%U%QILSN$p8{a z#YHhNG78*kp2hBTcCc~2&V>PV+GqewOf2xFPG}&H+!$a{xjG6-1@U?F!TKiCKc$Q< zP1wLX=65s{y}Y;)s$PB_c-im>HU_`uGBQ7w6OHFiQ{T5-&?>`vV%8fIP58EU>vtpq_CiZI6aX3w zEUdo?ex2^Rgs$7#c{em%=mOx8y#g5{nvHe$mN;#q1fIE)5)=64j?nkW!@~)_q2*~x zOiUm-THhZH_#psRev%xtWFQnQ&i+Y`p>L1%DTPc{3lnI3Y}_EP$x30khcXv(Yl7(z zdP+&ohH3q^=RSqYSQ61*k7sTyyxcN|03>UOI>Gp}&Lo@Km>>h~KWuPi^+bIf_=l0w zl=r!?*{-ynn(DGP*Qoe2-}Q~$wb54=G-?(7Bwv$!U9E7@Z8^?Jf}&}&JB{~d?Z7mw z5Uc=7@2ut4^8t-5;^B<5pPvvgVp9Tg5?2(3yM~i13^QsUd?q0O%x5>*#`; zkwLQs1V9$bv>0wz!J!WP2_GJYFIqYkzG}P&$-Wec*>sH@`>>+vznOct$*etDg)-?7 zL5;@~DiHI{=0bEN-I?=m7YJ0dSFUYkhhYJ~A~hKRJRVLK{Db84$2Gh1P_SjUe%62F zHqq*}Gpco=lNLbLV`)Y9B4sd=xkTk_M#wY0v9hM*(^_yL_4&T*r$@3KXDDe4H>^K| z$M3kr%&LM{YfF*}0H-6#iPMQz%~nxozvYAArT7{?rvC(Q=725%*I*z!+*u^6psT|m zWIJA@E{EoJ+}0yh`VJM=7cbEOF`)A?J0mTj=8e;Wt#r&M{w7+`Fs<8yC7-y)ht2GC zT!Q)5Pf5L~3W(HI(8tS2cGZ&fyY=|63MagB>F_pq(07*lVJL_9=Ad_Nq_a~P+$nc< zcG^znzp%vC0{*7XJAs$ZvJnV?jMdHNeQ(e)c_6i}2)P|TWY!;QAMo6dimHF&b6;gU zU1BC_mM`n_vTT*00Wn-pZ*R~-oFb8bz%u!{U+&4{fyNCk&L7W551idD>V-)F_9!~z zOi4=!);|nkIh6*rQ{hhXC}t|6q**LB_b78JtZ&k9Z#2d2>gvt;@@1xIL&)RTPha>> zbH-3kR@UFr5O_zgjivI|)mnZw`5mPdh33eeY}sd|B>FOWfdB0_`|uTXU&b~JklKv1 ztFG%ZS5H}luvOaAvGnWr+^{@TB$Ml%qZ>?iV>44`(R*+o?8=A2`r^K_0lppmwO@vJ zzoX~KF+aDqT3`X{3w@ZF5xB}g|7@1CpG()yr1#D>rkviu-k+NR!6PMKZyBY>vzG6; zR63f5^3$EUmJd7K2ZO=U(byo=jCSF1EN=iX;C0e^Y`-?y?K<%h&x#6A-x)C!baPlG zLmIbNjxzFw1UfIpU2Y_Gb&I0stRXC{GrfPXZG3(f4xV`FJ=WHEYC07|{PJ8#O>^7f z!T&IQMYK>Q|Mu+ZfYf`owj)_}P4eb6Fu1y}Nwt_pa%leGW!ZHcd_%DCF>ukp#T%WA zg~dMYBmUELoL+nY8Y0rV>eIOt+Erjq;j|VCf`3(BW+4RdD=)M5t;Enqik&4pLy0)zF;=XIGbHXsu7 zrWOr`gEXE*on7D45@l@q>5GVA8!f-j044|CPsRka$@xzYO@W7`eoW;Hu_W@X&(j3V z5ry@?8hvea8Q=g0oCrc@U*-I)-SdLVx7WK$+&3Sav8zKq$W@#{bky(L^ov*>A(w|) zNWO1R3rNSI+=hJ9S$c@c^7k`Dh{zlXuT6crn}gnO)uRW0QVwH$G9h zE8_J;)XNQ?H4NEHKV?O1yv%;Oija@K>-+NZj;qL?nv^uFqJk;Zv=#}MiU0Xe50`bI z-`ms1ygB8lN;V-L9y~n!FDi-aKVhUF>}se&Q*Tc_-2AT`FGw?QKM}O!d;IQ(*_oz^ zKZD?Le{}PD^YX8$qu9q^?hT5I+bzwV74khIxb^(Kht1^1Z2}1eW$v|7e8ZLRBq(|v zA{oyF1{RP4dWY+t3Ul`c=jP@ugw}AKH-~TM`!sji=IuEO%^Ufq2c>73@@cPc5cWR2 zzKmnP&l>qi*lJF_^OTAyKuX244?;Hv-rlQyeK3>T31{VSs^4$gwLNu^I6C%cUYXIv zwAyw9*_Ha_1>(=NcuF46t1;{G?(+P%*BsoQQ|#o}I|@k#9FA9T_Wag!a&q8Ez0t>f z&#K-No*r_^m9+@6h%}4WkYY*SrCe*=Jzf-1oAcjJ(6(NOZzJKd)r|S3XJiNn2z0#M zu7HbzK~3Mh8x!}=(OfB*>$IG=UtM3<3R0nt`A+^o8T0+|2@FLTmR&s$azbA(qWwmz z6E?wF^YZexx3|?R^)e2V!ThDEv2m08%a}k--okN;4Ann2RWuwYFUWHz(+8#tYinyG zBN0O;YVt=qBN!BA zd0!%=BK!qtV3swjEYCAhV-!LWE7e{0x%gi$<@ejYMpamdFw%X0$yK9qP*4!}e<3pV zyMH3z{bgQ5EA&(Uj`%iyd~!6*XJ0+~?DH4Ffx_bbDPp-OGcq!!PT)Wk|8}^dkB^V9 zr*?i~5eEMu_jwDqL*rz3_ag^Kkn>*}_l++kDQV$mBmF<{o;euriFQ1R{6%Sy{?llX z;5;}$yf0XF$SQy#b3(+d=Akdypdb?ylT`U%`N(H&U!cLGDPZj?wUxr)OXC+XqtSNM4cl<5is_rpHl0?Lb z3@k`^w0HGzU;ph9H){LKmoHaBhr1(52!R?^MljRNe{~{ayX!xGd)^lU|2EeRkuqYi z5nIX+M+~qQtVmAzS@h}^+{+WH(Ns<5>uagmGFXeW1yX)-9-RwPL=^Ie>co1d|?b|^_N^Ic8NMbub=)K{hGi|)YWd8zEH1{^gtGw9#IZO+fXtu2ndKpIk* zd1Cb`Ktw%`YFYioOxpL{LOzP7K&8C+GlnT0e41{)5@>Hce#bqBre>TMpo+)#+A=l* zt1VFA@-Bv@#_3ka#zrO##8iTQzLXNn5RPjqtqQ$*ZgI++UI=fR5l+(ZJ_r?XSD1#a zFRb{C_K4b)So1zC8VX>y`Akt!ru|_bbbYRf%n)7N>~ZAU{=6**RHCA!5cd9j*g}-$945UOeOzg=WZr zAA_UiVWzvGV{j_PoJ&e>GM-_pg}I`xxTB0TJ%mAqMR<^CjsjkX@^neZ9TI}u?g?B^ z=k!YO`Om~Qa)YDYhd$|!ljV?*kRxOXbBC(H&IgD0la`eq4qJA#>dE2-*19wLVyPys ztu$?2hHyO9D`5bCm4q`!HJCls5tHg8WHm{-s8uPn5&EU!T>EMI(koFGmxM+cT+I*G z78XBc0;xb%j;NR|!6xQ#EluJ$Dx}}imLbgy$SRH}V}Se$S;>8aHUSBMjHBGsHNFnB z3DbaI1ig0mBt}Y`Z%*PJU{4hn6<|vv-J38wY*IbUn9L!WOO_niiY$ako|Es{aWJ>4 zAs_Ci=kkEcWSXkG{41!{ZJ3_^%Dp6JNcUmxh0&I6S#}-k2n^55uo`J$qz!|C30+)*g z2$*!^Jhb=4BaGxF2JN%kJoal6H7gpELD(}zA9f;oHsvG6vVExJqZ>yu#F4r>!68CozOOkG`{`!etD9|zJ z!YvepB76jj$YD~ZhztwEpwhPPiTWr82^3e*l^N$3g|eL0*0CTnw9!qVz`zZU)Oe)k z@i@vo2Ds7Dvul);5MU$h{Q23pNFf&$4K5ze_Sw8_AP$ueAGt`wnb^drpv;k8T|Ys) z5J`|MZcfe29@{Y^*hto2bBjsNvjE(4jiF6RRhOJS{7P9k**9wA8UDvQElk@+tXAF|uOGPiLWk;utqk1tkhXD8N&}h^8=p!dx??`yI6Q zz%ViiamaV*gM?v2SDT+M3h9l8nO~JA3?Sg3fJl72Z-mVpZl}vj(P5@o2E+C#TOl#f z2oxelcra-vH!P79j1>K{!#I(NUpxe>u3bz{{uWdwr{{TO67WN151u@8UYw{h#3WeEp0Ok>Fk z|3mJ_LSdTEH54I}Cn|!l{tl30U-Rk{XE`}$aXt~0vN>seXSaZra#8igBo`0}YA3tR zwFmc_Y8wNzCxc|Xz$P+|93BE-a32~HM768V1ujM}2u7ydoL7$@%=P328iUpCUJO2O-#%~`pO?`IE)|eV zAw%Vj2f{%D{of<^Vh^+~$S>44jjH zT}y8T_u&BmC<<~#7?44bR9ewe)Y&j41A00weyytHlBve-2*|=BDd5Gx(J71Z1gV%J z*U@w_XOQOd(5YjT$zkth*~8VK0l;{~aw>5hHV|ngKBEYl*E7~Mjh(WvTYlwX#hnz|fg!@ORd5NKmn!R?1mOO#iW) zAg{C$H!NOMOnz`J?GFmjyB^x)c2@S9G3Lrb{%@#3B}V6&ZPS;-3cRq1$2|Tlv74Md ztIEu{>V_an$UPNyAP4YaTmk0~V0PiR^7tUPXTrk9J18d>8F zEtKho4AtJHjU!&n((<-=g7oN~P#+@{=SqUeyj+6DQ9!G?@kO^5@R1-lrwE2d2cPS` zJc1tkQ8wPDdb3$f3RFONe{-^sEmqO_iLOD<67Wo>h6N1X*K%(E$j`j@J$-8r799J` z3hQinjou3jupDtvutx>6-#h23eg>kyED(@Kfh>kqGz?1WC_=72h#h{k73k+iMPhfw zA7rr>P%UDe_FxDtnROfd*s~mtWe5Xqw^|H8sHXFiC|qa}^z`5{!F>cFok1e8>H$bz ztP~LqVf(lLRFfj1sQed8C43lm-I1R^@v z_7YWC-cWNV%vk77!2Ghk5>ccEe)Y9zWap5)WH6)1Hj-Ue%?BO zqFju{%z{W;pR$#g5HA4bWTJfw6gO}df#``KpdS*3^uIA-ZhcCcGM6}%(?4O$p`V5IaToOlD zNEUm#DDAL|GKRuN4<{;sUqv|wmz#`CGLvgnt(tw4A?t}4AqkgzWzya#lS2*Bxkzir z?*3S1-kjL3$en3UNSweX%sF)ki}E3T18Ml_fcv0v@9WfsR}DyA$((JEfZK`XE; zkEpU{7wbfF9wJzAB>;|oEH)e|T<$e;=M5xRAB<-MUDaS7Pwxd0fJ^{8&**ypwj zRf-?N`_yI>vV+`8C*hX%{@ptjHW4W7nhK6M1<>jPkLZ>kh4pz{4FDJ-r0|A6nQ`4|@n9vgffozDe6tP%_tboC2 zc77W_5x^ZNT+Jf8*iz6+!4R*hpNzf{^Tum8{~Di_ijhbOiYm@#_sDU0pNXxuJK1w7 zJvd!mUAwrd2q5+dv6XsDCd+PMN>bHgl|!v+I2*ZgM!ulD=)4`do9RVgw3j@x$VO zf$>;UbOsC%FRSNx`i3w<6cNc;ipj{~;WodF$&xaSVjzHk&V%Q2k79mg|E$a`ig9^N zoMZw=H{WP3IhPo#D^)&3;6=u?^mtx@+B!0(s4uwfoj5MJ5zP&tOVEYSLa4IwC8y;*n3*`3YCA18{ zQcuiy@~wu`qF!!K_+ZwF(v@U|(3o_SUAu zgN3v3o7D$;`t4*P;yyC;hHuR(@r9sr0sD=Y%P(i4XZ(LUq2{#~p0hkz1O?B|&R(~< zG8I&@>G2%GXzmzTJUyf%f6g|$HPlqZ?LgQ30HJ@=4plHg-0njp2hgQGLSuY7)Qa|L z+8MR$PmU`)e8E`EVBK11!{?4Tdf>@H+ty|+F1@l_j2&?b-m}Y4;%sdyB@3Ta401Dl z@T8UH+8-Kv3(h|@zsPMz`V*@W5J{YvN!cjh($utZ=LBY5f7#Fd=GO>ZX1!XADVb2+ zHqSYuD!lLgs-(4>5S_?5y2pi49q- z0QVmS>i3@choI0KV!&kSFO`G5F#dmGsqE{oe@PaMK~-g?ou8kO(%-UVXL#-H?H2ol z|I!@)nZSYlFP8P+h4h$LIMc>(nRLKO+=Fxth@BAh`u<&|d{plKEbqkM-ZW9ahK2Y+tF{ zydT&^(CO96gdhI`u^_R%b@!L7x95SrKGRI*)EepkPpZY&GuCRevb6LtoIvk3CmWlQ z(tZEoBhQRbX#rm_2F67S;ue)$NkX`hD;+#ANxzfP-!j*k-(Ke|U(G`Gk84p0Qc9Mj zpE7}LOIKa}tCZJEvV4oU95h`ly+28BtJI-~3t)(bg+GJ(mt=T3TocbQ`aXT?g%{_G z!0IQoc`qCFO{!wX?_B%S@p$qM?@M`x+D0Qa)rWQ!A8llj5{ip%BFe+T3#OaU*s_m@VpGhySeWSDQ3#fNZV@!UH=Tt>Bf#f3Vu-YRVVHCTu&ve0wK}Dl{w&9;2 zsLN%zkW)VRDE$uLzP-^#FK{VTBZcn@I2etbnEsTgdcI|pom51@)vwlpDc-^U-caF) zRnHTBh5*6&b-r5~0yF>tz^i6_QYzqp1TvD6GcyLMcT&8BSRzGJf1WS+Ze$)12NNlV zlL>vzY*CKUdXoF5n3|sZO~!~Go8YU6ykqR;)z%^&`fxarLq>S-VfrMunPdWMTfM)K z_W5G+hmwnk?SuH{--hHv;UnX6J8Sz(vf4?pxl-jm=3209bjK>!jhE=)$(#s?a4|QP zsZAK{!w1`t`U6t3j`S5ZX-X?JLyXFJth{lgM?#)23^)n=`xY5WLNaV$-FD277JEmF z=0+(0GAIfG%eVSDq(Z({@{LXie+9K~l#bXC{;XePZ_VHvlTV4TQ~GSB*I4?)Mh7S0 zbD%b#uP;B>LR3XH9;-fnDQgwfzDpr|iY z&YQu*55so|YLzRVIg}coe};reqkw<6y?xb`NuEJSsQl^UE969d;qSrjONt-+J8Cot z>Enn|+jlC?wm%usdPFK|-X+x?!UK`mv8fp({jCG+tGYZ-T`a$@U4QeHn4ZHYdwTd* zO*6ODgB>%xU#B#6%E;_}b>N+sOQjx}ni3h|P30Tx&8zZ<*xR(!v~{b4*TMcufK0W( zD?&BEV~#DQjQ8tYOdL7aG}v^guck@A9zJSCm(itUnh{SNBp@K5i^~SF?I#a2QtKX_3BuJG#1zO%YJE9H^1nBkEw41x&}Ssih?gxN14VqyBR*y}%6rYK|8 z-BM!W^R^`{dTzc6`yqi)ySq#LV^c&t)GwWU2?_WNs;b|^p&i|4jcndg4-E8zd-s$A z_W5n%W1=2f99_5+4cgMx;W_o)#KS)?3)4(nR=tx|`@X~y+@o~U+RGYr2u;+|< zysSbYkNyb`JReNigad@F2>Ia0VUPbH?hha&r%LQ{Pp%RqPveke7nS>TBR=^OpT6Vq`_S?&-VJ3-H*Nq*f=Ch`-M3|85E!vUdy8+AH z$v}_C8tMo9N#gOsX?L$Q6LWq+ciA8V_OQ{YJra=D(z`5tobg~3-P()N`j8O8{}qM^ z8DIgo#xDoct991qLR5*0{W{O9B?HA$F#78X;W*_1Aawt$8TLSq2LudO z)(N}fU+`@2mVn0!Dgi#Gs0avy)!x|UVdB*lUcT)AYgFs0RwJrCZ(DCd=*sh`Yixew zOK5?E{EC0*teYBk*2y#mMM!|6c^4D}u-z{{-7s5w*b~UNBe5Pr#*0Vj7LMq}2w-)z z&A}x@2mE%e>~lzm1Xy@wG(0IUx{Z(le+tO}TQUPH;`G^iHLL|UeE3cl76JyY1RALX z3J>|XE=Vkioq*w41=RPd_+pIGe?ISrT0{-2LYFEBKTxlBoj?KpSGo0OlD05Fp4ODJ zN^ISNhFXdVhn9pFd$6dD5-yREU@=afOteMpY#bI>OCc*S<^Vj<+1Ygo$c7$kc!hl$kqz-wZsE^B#vnAR{1R6lylrUeWrsnq1 zt9M=$?Q}nU(C&NSxo{EQ59$xD=2gw)4pH={s5sZds{^98vlGs+twlc{HC)=WxZ)I; z?)@4@;@0XGy{LU64%xU3WC9O3d9b1H4C}9l|7j#QBNR+4;&u%lKZf+Y0SpPI?xr>RP0_YrT z%#~}u_0cRro`zB zl;!_~SSv76ZPUOti}m?`F)V;9ArZnqBqcC( zG2H41A@@G?6r0N^xlzxXkhtCq!hgyzR&>CD>int5+S$>Yk z-aYbckn5Wo7uYe<>UmsC$5jkT_FY0dJ4S-sAf3mL;v0|Edn72}i;kXY`i)}-&j;S< zVB^j_Olhc?!F~w;S=t-Ch_A<9sT|HC6?2t`&p(B2v_?oDYgNt^ zw@~Gc>@i-hVINh2O)iM5Ivzl&L5GGAh65Z`BsBn_XNfzwrl6VsC@SI1&-?5#!!AD) zNbkvo0-Nz#l`ydsU{dnm=&HO9uG795n+PfrdlV``CkGBtQdO^n09%!cU5zCrU~B{6 zVHfbk&RdF;P8*TYWD>!m3Lj}(^CHiid}ZbD`dN*$TW2e~FmkrgVNrYA(O8bqSf_#~ zv5;Oi#VX2!H~I1~sY=b|_<<)`fR)2_`5kGoDhFL{J&Df7|HN0b+VXNg39XlrBZWzb zx8d)ze<~f6FWANJ#4jgKGf=Rm7=pG}AHL!DI?W98T?4{Dca#!j`L45rgmY;-L~|$s zsl4Voj*d+DRRiW&RJU1r1c2P z^K8t+nGR4<&;Q0ocmp>4YjVrt@ToT;`>}(zhVzuLA%shIu@U(>qQLeP5tom3=tnbn zS?dLsjDgOSfG!3f-Hcf9819lXBF*3#zQ`i zls+#f?p+u~8Bcr2N^X5jqY*(a4HgOka)wfOF_{$VI9z=E12s6DPIo>;H>u0Je)Sk( z$c(Jy7!F&C00&N+U31W>-8~db4V(vR&8%oA8iB-5iGJ ziI=gz6wG`KMZG0ok6SgLlzpuvG!iLz3IoS?g8PHr)ie^sz%JDHpGStLjhrMQXpr9q z%PxrxIenDzx-w-`hWdEhF!qb~`m6ltmi8r>$HtYzC~3Q&75`l0?wC&SSDQ+h6}Xx0 z_VycO%J`;to&`ZGJb$=|v;2^-E;TJSeI<4Pux!kX@BCRp^&zL@lc#2q!+8hzSBWau z*j~AF`F?9>5YJ~N0GXLcxT2%xH*@gRttWoC6C{P`1PN&}B_H=Ug=q(}Qxaw}Lx&e- zk@0OD;hf|@U;lnizfU!cAf$Jed_^UJ?UZL;q>DNEUikq6Fb+8xlQ)7AR{HX2-_9*) zv?yu_Z8x^!xqmBYyjuR?C3!`OfYAeBNf_wpWx5pEtW0%Baa-F=6kb8)C{HBBXJo%C zLMY&l!z92RQYZm{#jpo_&ngWH3V-NyaM+vAMCce1+L|I=0yw&-hTrn~=@L9XZuOx;pVUt1NcxC*AGCjP+lAq{Lmqxft=E{Cku zX~5H2lADu|^L_vv$*=k5y|>w~7Z(n0Zg->|7eOT@8eQK$e`+!hm>N+w`c&}Ov76zN zK|!4yEE6ca${X}S8IM^F4?G1M=8HUes^b-qys%xWHr4lgxe-GXfWFdudN^x2cjsS^ zUa>Mh&fM|FCP`KNCNYVG%IWCP3`Ot~W=j5q(A+1nT8lbtXgac0~ULIfXUp zHyW6FdwUxgY=V~&;zXDh8q;8fx3oM}n~ty`C+PjfRIG$ovO<5e3$IG-t$Y5*Qp#uu z>3`=m0ENf@8Diz%2ZmKIFE3JmBbdPPU-yrXyo`Mr?L>{?{~4y$Z6XkPv@F(ZiNg^_y3{Dv># zZ)7U9G{8hJmhy9>&e2`xL!QZ@kIA+g>1& z^Uo!5SZW$y^BKwoFJ1JxTLUthAZc;Id=8zRRq~NF`gNf1sVgiD!QpCfSrZs z91R1G>gR%d%C{VMiJzX_y&i1hqgQ)DPZ5D_-2@WArnlJ5Vb}Rf3y7rOb1T{h$$P&j zSWgiDE!}g!&8|%Eoo;#7{@v57W2^NJ4wLt*{hxQg86G)mLA(2Yq7JTW+J)@Xw-P5k zI+c`VyJ$#vLpFM|e%{+w=5}()*Q<8|GplX~4(D(rLW0|0r6rDfGTKNRU$fTc_X($a zAX^iEB|bq9yec}pc(>ggM%R$I-+w2$I&ouJOi%d}RCEV5UN&b~`EljhT%nz3aU_NC zb)d#WbG#^C&YEN>~6cJ z$>Md^l*Q)Oj%Dw$r|Xk4)!jJ-nD3 z5oj6_fXa2_0>=ZumS%yU-U_c_&xJ+hKtTv7i?5AwuR-4mjT-h7UpT~9BYh1P^RR}6 z9?>WF$=TO|@ACKdJc6a_LUmUM`$!8gY1&2tP)cm-h#()p!M+Z0hd;19g)Q@Os|;bHsj z>_L3Tck76EBz{IdJ6IFYfYW(Qd{0?xI`WYvvdv)V!HDo}ea#2}YQ|-53fCsD zGK5@e@21Lp-h-h~)6pzt9e>9da9v40&s{qRr!BZy~b*|9a z>aGXg_pd(hk%1EZt&0h(`ZBN?H>z;*WdJ z(RN+Azjh5N?TWrc`qCv|*>|tlGTL<{qrbQVp$%1!{6J*d&9+L&lebb-o?QQRQc+`9 zie0bZD9Sj$btW=proH|HVb|x}H#Vc=nX^i%ri_}uPKNcuB{Xaqb2(O2SHGF&p}STN z$~y9cwllksxiwP;k0Wc+sL8^4R`7iUZ{@(tO%ylvE}^ zn;LKya{860s^e`xkS8W0LUX}e@KK|*Ox%SIGb-Oijv2R1ieC-q)vS)}~ceG4y5 zcUX)t(Mn!u1MQ2|M;dorQ2M@_q_5&f{nP7DtZ&?%WiO$qYW%~$UgmrfvRZzV7G_r~ z?{?U0rjRT9cs5&3vkL&DJ_Tv=ztE(iM~71Ui+NnOS7sxlMw)Z{K6@9eDCK|MixHU_ zTzdYsx%*jTgfvQy$9EptKybtMk4Dq*`gRSeao59{2w(vF=(W07a=W%7J!TmWt~DsO znAfp*e(kx($1iFnd?>Jdp-q(yBoEkM6TGDzj@;;d#EfC0rvO0Wai?#(s}4a#2Jg@o z|N76_5SHNY9^F^_>%z+#asWs|CjWZcVhry6Tlv4{xh>qKLGVNRz9fT~_^e_5W6nC# z9W;}E#U3>VM-k+`UF3a3EM#X~=b@JKnRfKHb(1?mxa~?+&Z>)lqD8#T`qG^$4p0Gt_zoHSI3PRE;%2Q9Ktp^t#Vi*r_CmZ$(YGuY%{y(dffN@!+kyOf8hPY z`|rQn|IQ+Y zC%lpYP#ev=s{gE_8o}PLeS{8K@h?tY)B`SSPQ1ofRQ=+Z$Cl>~m5#OBChGyQH4bR_ zb&Cu3Ox#6L$oy$i7)icJ0$8u@R7aPIqWC3OG)-~ux~Ah@Gg#K@npyL@a?IR}Fbx8rRJ;pWNat_z8WJg;%g2LCh;L~lNynOWPx`szWP6Jv%PnG7 za4vdS!z^`yddx)aQp6=~z9jHm;Q3QeG;$y6z#HSXTb?VJRk#tjfs#Gaau9r z$b8c>Y}oyyy2IM;gPO)!MCccu+yydANr1G@My(*c&-O8qGy1(^+7gD&t zLVRDDAB!EyiUctEuOIRE7f0&#X+#)+LjOR6vb!x#o$0Ml`)al+zgN=ns6LMJflhO$ zeYTQx(L<0ld}1KsVbG@v`*cnMZNGQ^M)8&clz+=)UuzI$fsRa%`)h493H5c9-@V5# ztyG)JAfBGwmn|dTo0_T_+-d7XPYL`oGh!$01iNT%q9bn?UHtfkExj9T50ug>oi8xH01y6U#SzsGqDk#VZy{Eh877@Z#k-kmC4|1tj<&hmaf5dxTjwhRDu_ zT0M!1o!zGRTLQhXXTe6W)Y7M8nbN>yW7#H_tz?I|=KU?>SvK^xWM5ZyM5&vmQOD3t z6Z~VyXT~gkbB-68-@X}#k2@~bybnI!ct*W*h8U$Ro{jja+YgSIx`{($5Glj&L|7S7 zoaJ;E*)*jbuKt`grMhzct9Z0y*xkMB+AZ7Vr_!XixmWYqs`~ybQIc+)hyf;unMSbD zUued@FU}S`>b4^{ZE({8RFB$s?(5353FKj_s(@gHm6Cctpn6 zsYzLTADo8i+N{>!QHH0gSYl0DJlsEfMsI`^?))OWT9`94f%viBtb3JYM!z&ylR><| z|CW%KlkO||t>`NlbJfb{c~hh-T{ezdba!1VZh;^%IkK*<02fE*fFm|@nEu1#)9foG zq?#KL8#{>bBX4^t8BchKfnvON25x`KFl;?E1W1ZlZJ&kZbkmb4*vF-zjLTaV0ct?$ z5{T7Ad#7dM1I9e(l|Y3PA1wlX(9hpIhO&RyqG?DoSA*qUUL5!A)9FZO-yTetwQMEi zeZWS^8AS;#=_aclPbbEl=A0(q6~x`gBoCk23rwC3c+^wo+W@Bb9WfuQWa5TY*a~+s zx{zk83xTve9#gutByK+BKn^q%wa*7Yy)%T8W^ny?hHzNQm-Cc)8JGrB5)eP}?&~%h zOJ>|LJLxZl^{O8#IS?RIf~9e0VGccJUK4q+MDls_VXX0C3scE$v92UkRGhk>N>D%b ziMHM{%Gj8D6J1zYm?FNeBqHa+j-$x$NANzT-yOA#W@uI}R zOV-{~K2TFR@uo8ZXNRY2)p7Ik&^6auK2P&Y3_e|Fgzq&o2!u(XnpfOa-Isb6K52kF zPo^FU4(m<^(rCzdfD4Yf>IyK86l%3J)DMt>r?T<8Z2C z#vqCQQh^1GM!gdYf)i=JPJq7uUCZMB3&)M`hs&)1Kz@8I1WN}MH_GfUMGjaGExjy} zk!ag``(|Y|63FLk@3CxVyx?X@0MfX-;w&B~>X#r56_b07qCJODZ+-5}82X@beQ-co zda!F5Z4Cjif*D=nmUmKTkZ1A-^Ri{%_?`0Ttdx5P-kgbDURI!9T$v-oypPp}>#CoI zS-nl5(x{h@FVqFS)odtx#vkp*xJ7du&TKmqRN96Zcg(cwQ}IvWnkRg2SS5b+Z81D~ z#N_^W;dOD}`hH$lf|3`ZfvywhM+Q1ruBtYP5;tWaAuh{Yk+ws^S-vv_03ol^jLe`o zo~JL(GD;Rmv{7p|829 z>yz6P22)70Gl)62`H0-lxLs4Zws!96*iC7p z_otxWhg3ys5=I-Y90TJY26J<_lRA7DqiY3)NxOhvVTp4h!nf+WbrH1Tm&t)AVi>>6 zcjk78HLmBjtY3JlUE_u{ z^y4U{P%Cl%QI?m1hh-#n7^&k?KVR?Dfs=qrI9n27+cTli>kv(*pyw_LCS16D-Sc=%i{#NV3z+iZ_m;BWX8f z!O_04@x4h+)4`XMYcu~!Ii(K9u7eZ zQL7(OeJ5DZ`LYyywzl=KInw^)^gFe@nlrkBvaC!N9Fc=T&D z@n9=0ttAIDOgwzbze|{dEM71z-XgG>z29d0=@l@UWLHstO_u3;(ly($493|D>AF2V zy6l`+GqDAYY0K{u1f^;1qhPay8ahqEMQ4KaVo%!vg1h4PIJ-7QP7xZ@MpQbt5CSl{ zeHEHw+;C{noG!YQUa=wNpqQ;?W;i#%{RZ@Ryfcntst2jM=Z$)#CHsl^VeLsVz>*ouPt2b~-pL1Lla;0T8tR&SlX zZ!6gXq3XbbjlTC#_@PsHKW+J8K{xv-xXuxHLssmmifI-6ar*ZNz*mK2S-P;g68|^1 z>;uqnd6xmgE@cNSs(7&L5MoKvj~Hzb!`>jt^j6$ji$|_@GvfP{cKA~7eq`o>fPX_j ze4KDG+PG^sfh4m7^BNU`3nl-;opFC>o7LdgWTH*Yq|J`)6W_6J-JnE!q1>tU7buO$ z$;r9=U4EyC9Pt16=zlG!|B>ze4+7m0`Rz;3lNN!`6d3uff*vM_bM=4==Pub+pY=`o E7xvXpMgRZ+ literal 0 HcmV?d00001 diff --git a/doc/source/autoscaling.rst b/doc/source/autoscaling.rst index 570aabe32..a882abb75 100644 --- a/doc/source/autoscaling.rst +++ b/doc/source/autoscaling.rst @@ -1,7 +1,7 @@ -Cluster setup and auto-scaling (Experimental) -============================================= +Cloud Setup and Auto-Scaling +============================ -The Ray ``create_or_update`` command starts an AWS Ray cluster from your personal computer. Once the cluster is up, you can then SSH into it to run Ray programs. +The ``ray create_or_update`` command starts an AWS Ray cluster from your personal computer. Once the cluster is up, you can then SSH into it to run Ray programs. Quick start ----------- @@ -9,11 +9,10 @@ Quick start First, ensure you have configured your AWS credentials in ``~/.aws/credentials``, as described in `the boto docs `__. -Then you're ready to go. The provided `ray/python/ray/autoscaler/aws/example.yaml `__ cluster config file will create a small cluster with a m4.large -head node (on-demand), and two m4.large `spot workers `__. +Then you're ready to go. The provided `ray/python/ray/autoscaler/aws/example.yaml `__ cluster config file will create a small cluster with a m4.large head node (on-demand), and two m4.large `spot workers `__, configured to autoscale up to four m4.large workers. Try it out by running these commands from your personal computer. Once the cluster is started, you can then -SSH into the head node to run Ray programs with ``ray.init(redis_address=":6379")``. +SSH into the head node to run Ray programs with ``ray.init(redis_address=":6379")``. .. code-block:: bash @@ -21,22 +20,70 @@ SSH into the head node to run Ray programs with ``ray.init(redis_address="", + } + + setup_commands: + - test -e || git clone https://github.com//.git + - cd && git fetch && git checkout `cat /tmp/current_branch_sha` + +This tells ``ray create_or_update`` to sync the current git branch SHA from your personal computer to a temporary file on the cluster. Then, the setup commands read that file to figure out which SHA they should checkout on the nodes. The final workflow to update the cluster then becomes just this: + +1. Make local changes to a git branch +2. Commit the changes with ``git commit`` and ``git push`` +3. Update files on your Ray cluster with ``ray create_or_update`` + +Common cluster configurations +----------------------------- + +The ``example.yaml`` configuration is enough to get started with Ray, but for more compute intensive workloads you will want to change the instance types to e.g. use GPU or larger compute instance by editing the yaml file. Here are a few common configurations: **GPU single node**: use Ray on a single large GPU instance. @@ -84,10 +131,9 @@ with GPU worker nodes instead. worker_nodes: InstanceMarketOptions: MarketType: spot - InstanceType: p2.8xlarge + InstanceType: p2.xlarge Additional Cloud providers -------------------------- -To use Ray autoscaling on other Cloud providers or cluster management systems, you can implement the ``NodeProvider`` interface -(~100 LOC) and register it in `node_provider.py `__. +To use Ray autoscaling on other Cloud providers or cluster management systems, you can implement the ``NodeProvider`` interface (~100 LOC) and register it in `node_provider.py `__. Contributions are welcome! diff --git a/doc/source/using-ray-on-a-cluster.rst b/doc/source/using-ray-on-a-cluster.rst index 4e57e3bda..7135f0c2f 100644 --- a/doc/source/using-ray-on-a-cluster.rst +++ b/doc/source/using-ray-on-a-cluster.rst @@ -1,6 +1,10 @@ Using Ray on a Cluster ====================== +.. note:: + + Starting with Ray 0.4.0 if you're using AWS you can use the automated `setup commands `__. + The instructions in this document work well for small clusters. For larger clusters, follow the instructions for `managing a cluster with parallel ssh`_. diff --git a/doc/source/using-ray-on-a-large-cluster.rst b/doc/source/using-ray-on-a-large-cluster.rst index 0d31c4c9b..56e2e02b1 100644 --- a/doc/source/using-ray-on-a-large-cluster.rst +++ b/doc/source/using-ray-on-a-large-cluster.rst @@ -1,6 +1,10 @@ Using Ray on a Large Cluster ============================ +.. note:: + + Starting with Ray 0.4.0 if you're using AWS you can use the automated `setup commands `__. + Deploying Ray on a cluster requires a bit of manual work. The instructions here illustrate how to use parallel ssh commands to simplify the process of running commands and scripts on many machines simultaneously. diff --git a/python/ray/autoscaler/aws/example.yaml b/python/ray/autoscaler/aws/example.yaml index 933d3db8b..82375664d 100644 --- a/python/ray/autoscaler/aws/example.yaml +++ b/python/ray/autoscaler/aws/example.yaml @@ -3,7 +3,7 @@ cluster_name: default # The minimum number of workers nodes to launch in addition to the head # node. This number should be >= 0. -min_workers: 0 +min_workers: 2 # The maximum number of workers nodes to launch in addition to the head # node. This takes precedence over min_workers. @@ -70,8 +70,7 @@ setup_commands: # Note: if you're developing Ray, you probably want to create an AMI that # has your Ray repo pre-cloned. Then, you can replace the pip installs # below with a git checkout (and possibly a recompile). - # TODO(ekl) update this to a wheel from master - - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/f5ea44338eca392df3a868035df3901829cc2ca1/ray-0.3.0-cp36-cp36m-manylinux1_x86_64.whl + - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/b6c42f96beab3ee00fe4b246e5e9d0479ad379ca/ray-0.3.0-cp36-cp36m-manylinux1_x86_64.whl # Custom commands that will be run on the head node after common setup. head_setup_commands: