From a128333223ba16292f7c4caea421fbb39b5f755c Mon Sep 17 00:00:00 2001 From: agutta Date: Mon, 19 Dec 2022 16:07:41 -0600 Subject: [PATCH 1/6] Adding new file FAQ and an image Adding a new file to capture and address frequently asked questions. --- fast/stages/FAQ.md | 23 +++++++++++++++++++++++ fast/stages/IaC_SA.png | Bin 0 -> 17535 bytes 2 files changed, 23 insertions(+) create mode 100644 fast/stages/FAQ.md create mode 100644 fast/stages/IaC_SA.png diff --git a/fast/stages/FAQ.md b/fast/stages/FAQ.md new file mode 100644 index 000000000..7b3832ba0 --- /dev/null +++ b/fast/stages/FAQ.md @@ -0,0 +1,23 @@ + +## 00-bootstrap +1. How to handle requests where automation, logging and/or billing export projects are not under organization but in different folders. + - Run bootstrap stage and let automation, logging and/or billing projects be created under organization. + - Run resource manager stage or any other custom stage which creates the folders where these projects will reside. + - Once folders are created add folder ids to varibale "project_parent_ids" in bootstrap stage and run bootstrap stage. + - This step will move the projects from organization to the parent folders specificed. + +## cicd +1. Why do we need two seperate ServiceAccounts when configuring cicd pipelines (cicd SA and IaC SA) + - Having seperate service accounts helps shutdown the pipeline incase of any issues and still keep IaC SA and ability to run terraform plan/apply manually. + - A pipeline can only generate a token that can get access to an SA. It cannot directly call a provider file to impersonate IaC SA. + - Having providers file that allows impersonation to IaC SA allows flexibility to run terraform manually or from CICD Pipelines. +

+ CICD SA and IaC SA +

+ + + + + + + diff --git a/fast/stages/IaC_SA.png b/fast/stages/IaC_SA.png new file mode 100644 index 0000000000000000000000000000000000000000..8247f429e61530c4430e4ead35921ee08f1f6898 GIT binary patch literal 17535 zcmeIaRahL&`tD5@IH)3ouUspuo{d1fp36^-K&eLHF*3W51I)NxIy_MgI|H=O@D^y#rLN+`A+ z>c3C;^jI#V2mceoqC)l;aqq(=z>%+L<;Vlde|`B^LE0>jh*F7~syz+=DLNE;|Gx$P z|0p8oMR#wnn1%*1Ha4~l#qjh0siiskV-}FSXv5Pq; zciE@J#KOwV$x%P7w+<^VX5kdEvDt-f$jHh%obTyz#^2}?vuJ*u()W-TWJ_T<%{gRc zVJZCGRC~!iihof4@Ai8A4J9-JRcby5^e72=vcE)C)_jEC=&~R2iqG~n56?>i28HgK zDl<`|?=dlc1*&;)@gVH!2FE;CVQOm87kFr0)fQuac6V9n%zr2IWW*-6wzftkG&pX+ ze0(ndCY0CI^aNlK|JmGR8a2JYIgX6N`W{Mb;E7kFT_YwVgYJEIX-3Tn)hBZq&6YMg z;+7N<5$W#h3pF-h6crT>4h`-48OI!;iSYFD^5Qh=e$jk?cWoqSIc_&!@9+W*P1N4L z9G4Y8m)~){9f43D%*nkxXf=FjbdGP5ihrVm6|_Xxv)Ya%dHr&)D_wag#w40#+z@Ve zmm8dp>vE{5sDD;^LN!}G8=@i=QbEh(1*-2|_Q>;bLKMu+85g~;ZKBU)6cl==41B_S zLWx&THphY^B8YDDDFj_{Lqb9ld931bDe$v4(mWKvy6eN~?A<+!iq6ZYg zhAK^ef$0L^G<5GyxNzlaX&ixK2fVT_}yPQrz?29y(? z$7Oor11({aPFM5-Wu-;Tp$ajGcyIXHAnF?7XK9hI9-V4az7WZu!i=r2{3_wrOnMEG z-%*oQpi(4^jiRi4d>Stv4`A`Y;+59?X{c}rC}eJ<*7{8XBXW>fIoVQBiEy%~e0+82 zWOcR+_y}4UZa68I1lVb^M2CTfjVC1HhJ4daMqcQ6kW0*8%3dOZoC%xu5T1{d)38Ph zq5~-_E0YK~Tj#v2sHniJ`JlcrR%4~{Mq}>~6b9Ki)ZSlru3-Ve%3g#)Li4FgtAX-*dEU)Xc5;_nB0=Za?6yb0?G9e)) zUG9&jg%1U)e8eDT>jnlNq!Ff~LB~p`7~5DV4`oCjZ*s9yHU@9dQm1ej`r@WW3SYfY z6uzX>)zt+qYkG^Gu*nIMv^g&5I}+jzszeI%$}X_5$3i3zO$`yz{b3MEntXStl{G^7 zvt?`}yfM|VD=;LM=q0X3OUn-$UbQ~eDYL1l*r;r!pEAy}_0|e>B2bTp+}zx33zd@~ zTbD8mmbXf|{6B3C8K|=wQc_Yh$gT~Kqq#%+stH1ej2jPruT^S62@PI6G=L_Q&bLsq zZk+|x6@*jP7k8M+8iEq4Age7u(L_W<(KGc>9WLO^j4eb7KMXr3hY=*k!#$sfg7*or z>~+8htV)au=?^EdNyPuFyR(qvt6wfgh-AJ(@&ee3VY)=25i1xyaN1~6U{olbB75Pg zIIf=NF+d3x@5VjhWG{d+T3LaryfgmykkN#0J&MG!dH1l~1STtX!^d8Gd-hEM3`glW5j!^#b@LQi%LZgo{&?}CY#)2Et zSR`!6exi3>UAE6pE3G6^F4?kxBKK#DM&o4hQp=g6ryW*M*CWV7CW+!@3@8VFFZFfV zgLsJ`Jw4?-e0>*bLdzz`I-KkK*XPSBVviR|1c=T~-)l@rS0Wi(ti#mA8 zdbr0Kkx3kN@BHlcZS5^K=JOq`O8u&kQo5u}c&VeK?>z5*5z#Hlh`ZSFA-617*DlSG z+nb*{WC=3!vRDtTBhC1TQu`biQpVJ&LRflnI@}V;@5 zwHYLNuc^)C6%xcAfMH0szKy|PWwirxBIDc7SM<9T1#p^Uk2wzs7cVLBDzyn#T)&M3Mzvo<*F)kMrKEmk@QLhGGY zdDoly33}-zWQG;haZC`=^SjJC; zPL57HGYOVgI0CKAEV==yc+gstN5%nV(Th;iT3)m27VnA(1mmlU@@ymE?mKXP#F zdLiPIUsPbY)KK-Fh%{uh@*UL0si*&*GC}XpPz-}Tqab-%1Z6ny$4RN*^+EpQYOehJ z;s%n>4Wsd<@reQzhK7RWr8`y%#>iM&dhk*u=04Y8%fXL3YIV(H42lXegF=Y{p$IXY zy604Mg~3R{^GRzGGSb()D{q~PmqL|Q13PQ{Fovjb`UbMhlw%EEdsDtIEW5})Z%M)V z<8J*J3U&IkQ1Zkb`#@aT5btim3AG~HVPs_dgCrIE$D%c3Qm*jp%M+fk10_1R})yx(h!pBd&y;$T9dLsG{ zqpPB%C91fVa>#tCf#S^O^7(G_{6Y5zpPNP~kM&N;@>mrH_MR|}zs3BZGFB@K3kW}2 z29E+i>=@5M^I!#aS}VnPY|)sJjfv_0&F1d+|92qdO z?*3e3(O}*lZjZeQ8LGSYkO*r-HaRh}GIf3D%QNoKu*%3Q3g=r76LtSgWVG^qd19zy zM$+j{zPvAl&Nzg*b0HhIK3xZ0@wtToGqq`R{364$0-Ay@Yaty-JK2(BRM{`HluS6u z2xYE#8$AR&r?iyr&C2ZWlwf%U(a{Bg#PK3xmC;dq!MEt?zmr)Pi@USwy5{>|#7o58 z;}8-Nuc|=_6?(g0U+(Ea9UMANyl)}UjVk#R4$y0qZ)y4$pLWyj%PTu~GDNW|<$M;L zpFo|+Oo}D>rL;)xZ=KP029w*6bUAS=)}G355Y*1OfmOEZUx*gmo$(C#9XG8}t+ll>JwBLO>!0#z3(R!=9(%dlte0t*5jn9`=2x1= zMlUS%5wp&7fZhI!qrZ(+N)QYi;=9h~Sxb}5eVKGv?ousod1P~_EeWr7n!t9jW#D6= z)0G8lTaxMPpNRR{5d%}5R8H`afrpsGi5jZDhdZGzxi!S(jSwvc+JFm3z6S9L4w4QW zgdV#gB0%K!^l9Sx02Oj&hDcG7gz@vycJGsVT+$%+#iRg54Qryv2|~oB>!rkevlq@1 zySKuLR)sK0LKeZnX(#`CH)^sJzpn1jU!2P!20nIB%PEfiBf1rp!dH2vv7Nood0sz~ z(PN#7rv>-TOHt&4wn%0nuY~5TGQ#5m-^aiUigY$NHGS5C^Q)LxS$h__%>0p2)ltn+ zNTn=3NbJz}d&+78WsPn}Z_2;vgDtnNBw}m^y*dQ+e>B0UVzjh9tJ=xQaLuIIpfR)d z4|Rbn*iZQ#wq!&y^I)EgN{Fz_*VnyW@_}WSer^%OLUWL& z5hCPn6lrd9Y0ngrgk*l1`Z}~C&&y-1Y77O|my)%);AY%_>0m2HCbo{jguG8<`*Ywc zs~goU$P$%^UI_2dS-F+5hBUNQyB@Q2Hx{Z!3wX zvIq{(8}(@5tH>24QFac-j7L%*DS_CWMuiDZ;7*VK%9)}k_)eIFa9Taglb`ny^W-xP z{8ToI*_F+ZC!#&S%DkHKZ6j(E1+s3MVq5iU2pmERF5ElDTzzJ)ju?lFRC^p9iSPY+ z&pyVfXuLycUCk{?$u#(tEqak%1_bo8vxx1S68isWWC2?(av3k1mw75@gXK=QfcT2t z)6@P5?rX&5$-XYC(|GUK2<-!BMjz@ZI~Z(eW?Iu<0{m(5;_y8=Wkd`Dkw`-FLeIvg zSlAUe#^cmC`r9M#g(l({_w&Fqk0#idjUbut7$;}qXdaIB;9&Xri3OORe#Ivyy?eB+ zj#hvjM-hv~Cx$TLyKhwRi?G9U$&3V-7K?J^_^Zs$Cs9bxY9Znl!D`O>Dqc_B!hI14 zILH!;iFtC^S1F>bsp-pKCNH^CLuaj5MIii550P2*+=!;Mqo<410dW`{-V8fb4DSrz z9++;RCRt4*dqdggN&h%27fbS^1n-&6wIm_G)wa3udG9Aok{M!&+`#Etce8MKOkC0% z6}3e@f%dsR44<+EtmkGPCzPWE%x&6OO}w>8p$oM61~+0;)0~5ySy4kM|Rba?jwnZhz`NVlE?dTknbu-KC7y^mn+ma zg6JZ3q{k*4b!1&8%8lIgs}5t?0l79NBg050pI*MLwlEDT z$Z}s@3m`$I;tw)QC#QpDrZ^fSEQX4XF9yR^DAdLcbX3)#)jT|hvi2PLi@l4ZD&Q}^ zjnIyaJ>h9*7PaiMd?%=LZ%oT|aPljytm=vNk*xPVyU9`(s*m1HoM@7~TFn#k;20go z@oE-;?VaSBBW6Vm;dzmKz2QaYz7-FBqw-{pC|1-U?+hzHAF0Gm4-zD5;(eAi9Q6UZ zJwwWXlipDf#RVuaz6AjxjM%)qT)lg3zR^Ei2^gOUf*OqvrXMk~FycE2P8s8as}WV9 zK~b+@uWc_(t8fV&t+>|ahXec^i!AXdd~qGv@Pp3$;ji0JYR@J~4%cw;AdQq1k+%Za zv;thRgdff+$x-(A^kp18rbYj?GToA=3Y@>IFzi#-m2VFD{#`|xS-dVelKc;|H=g>- z)Z!ny#WT(%$_{+pa)Pt9L`KFY8SHz#S595tYk!H`dl1-swhIB_rhkprCY|H!9gf+G zS>+GrV>`+@`Fi4tq&3@^Q8!6vjEtW_PusgQ6+h^;xDn@6BBkcT=69puWr*+WGX&MNP81>_>dZGnQp7A|^ zTGF+{yI!p-1KOhU58;Ol>9;&oOhNK%t1S&Ogblweau^j@lw|t zU=JBR#(&PHQ_#t2tU%_Ee-WyU`5QcB+;sALrSl=5dF-fUmEwk3?ii3Yn`5b2Bqz=+xQv3?^~)5EK*?T-+QFbAD<^ zskU39R8aVR-9_^X6HI^!=`s2&dJH?Xm3Ypy!VotYwJ3BkMWNx(U^ZmY50T&LNS#F03H#axUdVA^z z?XLNJ(L=M^94{;^%>LrQa0gf`K7alUBx(4Sgt)k`X+YG})Gx5HiH^jSlm0DG;u&~f zzOlBkaaitr>gw(ekuTdXPvtSJjI?qCFxNaAraB^VuYIs9)&v#H#gx_nL!_*U2Rv;c$&+jN##ffjXJ$pa#6nma-B|O_p%e&gXn( zlTNJf`bai`?WM0ME1;70V@qR&47n_MF^r059$fbhR!6qhfai;9~dEGEluH2Oi(_isu((wLY@vJ&;C$=6$%@?D9V*5?h z?^y^EuO^h3ePvkW9z0d52chemn5e-Ap3~lWPC?!WLn6ZhO%NYpF)O=P#X}Eh=$@w& z8Z?$F>QE_P#!9hpu-QVR6U`Ql)e)S4fWVJJz&TiLGMUHfV6SN(X8R5Sbp8AMp+Fi) zr3602f*GHpFe}<8;%a0;bIJRf^*=HtY?Tp4KuH;4+!qnDvJyhT@KLZ2K9(pD*!THC z?B-?#B)#0f3oAnsaB=07Oz|M}qzs9tood5k(Zp_u%<-2K3rgt0qw2AZ+O3}aASm)H zAiRbeGkKZX+D6zeG=^xC0jJRzf2WtxLtM})#2Ud58J!@*ztcqw1YH0x0goy9ohiCh zCtzUEhyeFMxZ-%cE|2E`k0lMknZQK|N>?)eg)D&v3yO&9Y%YBPuf{bD(6a<3LKjF9 zWZ+ybd%CF4p22*L8!6@ig?w1}x|mTSo|ppUBO?RL&&g~m$tOwj_oI1V*_4Fja&+FC zXR_VM)usTdWn@ygFw^4U(vzx6$3uw76d_EVsj*7PU&`#q371`sDen)(_BzAPsmjk! zT(g)iH#|5#CJ5*q7Ca8aWrqm@kFEmP9htncvI1_w<@>j_q{IqnYK1lWzin$P$6rEJ zkqS((z~HIA0JHVEAT`=F0W_)x4i`%&d^5HgsNG!xYqvLqXF!)uaAL4gkO8~lK$g@ zaXgx!0rXq#?Vc*re;FPR6(D>K9718)`}tq9ET8BR5ygZydm8IMrK13co_=!B_^-?M zp$6!x{7Lsi>wijr`k+Rlgf_kVuai~?0oIk8$xyRj|1F*IVL^MOIUo05Q}2HoR#4}k z(jNhBjo!Xe`p?Z+0-C3C$?3|>Kc%DN0*6qDT>iNkwNRj4EK7)oa%AgI=6u5*06I%oOMcEPs~4i21v z8xpwK;=x7r@FaYdd5Dw-Osx{7(*@n|0oO^)#Ds2Tb(Ir}RN_1U*6z&AOyyPS)|(^Q zFftx=TU%Q!E;Py)!*Wty*n9rU)dl{+mlEPhIUcO}Ivbvf(F)2wM>~AtW zwnR!wy25W;DE>hoYGS3OrsjXWaQ8#)e^`L?{jSbVG2IssFY_|M($>&TEVldnIXCLF zxOkT(v7S?z(sNM6ol-VpbW~-Bb}KI{>j^DwmW=U;TUBM{+uef3GaVp;laZ16WMrhm z`B77oM9TT|?__lg_A#DOx6R06#6`_-NvDr-W{+mwMHEhq(A?$Sd+Tj zpDs|p{UJ_6Zf}e0L$N79bfaWtEoaHDhT08Za$H5Th=V))p3`!0;0n0x5e~m$V92Vj zCY+rG98yhJHRIKRQ0$Vwzf<_4lapb-zK^LXV@J`kv7ZenrtsQy3saN2{*BZ4v%MW! zq)~>B2iWWc!xom7Fr=Zp#~Tnxdx!WtJ!l_@T`Js;KUbJZC4yY`!hqEm-5$97@vctt zlT)wtB(sr5r)}-?S2@~1fWQn-sFt*#_QC(W; z=+ID%T=^9Gvr`~q(OC6VGv;xNm-}-_H}VIr`ti)w*E&LjmDt=3c~S@)VsE>l(NAOy zfY|IWwygAQ1`zA#1BP?L$cPd(e`8~#X8EUYu8Rl+9uSva0LJ0r(J%H;Da7VjUeEE# zkn8Ez1mM6!l$4bGUNg6(PrP2uzk0pZ8<$0acRqKXhv#re z=)q3aaOdKpem~1@Iv%Y=a6yz@)Tb~?p~lIUFZ+u^ojfM)J~5Hx=4{UH0+|)rtzTl(hTsJ93^5aPOc@RNq<2pl z6n?bFF>8RqL1facO-%wLef`M>{)oYf6-|hGk@S~}#>uan-+W8m z4C$UKc$IlB+b1i>6B-7~+huJ;0sSXNV*B0{@-7^S8G}%fy2ul6=$=nJpa7JDLuqr`uDZEgUBO zxQEC4ZTAsC#2M=KVYCUlse<=3J3cm+&LoJ2wr2OGOn}Wy<(ck@dkalQm}X{X z)bI70oL`Ea-kvY{p|1-G2{D&m9uJEUdxX*O*6i}(r_xGB1Vanu;D)_>^PQ!$+rPz^Mw zpJs1aTH4om9^6+n=sgcraMX!@mK<3L7vKN1w@lES6F1@MQK(H*9j4{1*H-FHY3vXu2Zg^V#b*ckB$@Z8s?j>_8Z@SV7Cug{5P zr4gXFD(659R{7x?@^R@6JE&$iE;g2MXR)PeRG6j$E``1O>g^8(G$0mjkV;aHg-otN z@^Wt_zDVHZKg8<&{0k0J7M$*K!f;vw3Ve^5yFAB{woPrTWJPgFs3sJJC^hh6cN%I8 zOnMYn!ZGBJOd&ym8RIW=5=<U#_PR}((rsijG748!4>N=i zt$)t0jnzK|&P({xb2BHwQciE8dWrFIwe13kMZZ~dFo;HyPLU2oxS8qB_5JmOVNF=f zq5`>2KjWDwp5Ug!&j_?;0kMFv8q$-ESb%6fV{jv1B10H1zKLEan^8^BVm^YQCg0Pq z7JD^gmL4(}SzsT7Dy^u912vQV@ZoFlijo6UsKTo4$T}m)abxIZ4^r)u9G+<5Tqg9* z#5f&2{bapHepxe8X$uDPWzvHj$vFvdnGF$gFp-JF#hgT{&(}hut!C}$mPKw)VisHjN%4&sprm*D5;Uuouc0i}WS08D5Ny$T65@|w3LtU?o-?E5Y~ zOCPTKO1SYlAh~!*4E^(4#y6aWyHB3ygk?%mDd_RoZplQ~M4jfGFhK4Z--!im5{5_L z2S3&lDewqD^;eNpKc|*{Azed5r7w*~I-A0?u&3b5o?Z&CkRDv4%4(S6r83o=fMSMX zhboBg=H!wELGlWuj@{nkDbc~Rej*E#lG57y|0wxg!T9jHo22hy@bQ=v$bs`FNoeH9N1^{nX8HLa02a$YC_j0TA z;cBtDIEdr~(YE0+D;t|{Zf>m-Z*MwjPAyx1cXTpzspib7dK6@L56$jD}9 zmg*A=A2!o?S-mQ!?#;KVWC1zhsg!I`#UZ(S^y3=wZ@zs~k>-i?CP%pSgSz_yX^%W# zRK6QCG@L-+Tf8%0Rk`CFC04xiyGX=9@<7Q3S*&}EbKbm}u5uu4d>--D#)_D5ZDVB+ zp;zy=`j(rg#?WesXd#W{8LJINx5N*z{>7S=_X5vd5_M)esywJjNK-UY5G~a@FJ_L- zdRAu}-ODR2{ID0UJ}kAruqcq8u5e~*6yhYO@cJe4^drJ~x!d~()eIrfPYbObSgX8hP4eN*drU$i?n`;VW+SdVDJN}2J-K#WwD z9g>$*l!~1l6?y=M9%0>7n?)jxtv{hKFf(JG?>>@ku#JVA#ZiXlzZvE_14xj-|_#qSw3ZM4+qs?tH zZ=Mro(PQ1GJo=p11bn_Vw;({$`2=M~*5r}RWe!wXg%#$E4r1<%eV znJWDe5Kto|q2NK0pg`5R8AYKqHS!^)cCkK*kQQt)azXsbcwg7zW1Za_qm6Wfj_E;k z-jZ4PYmw)CD`g*+MhLTVTd0Se4!gbUtM}>Gi{xU6DP-q(Pbtev`KwV815AuvlQ97Y zE>pfmHouignCcS7*bfwW@16oa2j=-H#Qh`Dov_2>Z)s^Hn;VfL*gKNr;jb>Blh)bU zdCNs5Isq?E4-VSny20e(!R=#!95QThX}YiPD!`xF;`c<1h9)XHHN1{q&VRsXX%oEt zWp$5&WuX>JPgi%OO8ohch>(r}qL8zZytiq!{8LjdvC9impr?xeWC4!LAVcqVR3E){y^BcaQ4*r|Go>2Z1;G z!3yciTz^&&SXxOI(^q|Y92~q|7g~P}$QZ5OAr6rwU(-Zmy1R%K$lGT|^pHYOGHact zm7kYc=6|QLc}wP2ZYo;Js~y1~d-Y7rax=p{7^&e`n74WTO6qyA(r#}8< zZvD&+L7n+M+-!dwdpQqVOk5R%o`_c}xZ#DDIDzZ+$xG$88S95zvq)A^=xEhXj?Jd) zkqH&QlIUlFE>jX;o1PWRQA$M;A{uaAu!?Xf2`g7-a&?oXxhs#44^8%k+{Ui7NO$7L4Ty&_7skuo^Ak#SF@pzyPJn`Y`Ra+cU@_jZuX82 zvBE2-t*~a_$9QkPR9AcpFCdlq_4DAavBuoWn0ApV5{Y=y($c)GtwKbRgT|Bo#3gm0 zr?5+_yl$lP&yf3p!^F1$Wz!H1HK=xy&ZjX15LSBYff>E#=Q%ou4G4qx1YC&vXlQ%V zw)eK)cBfG$@Fdx{f!w)gHMMAYRw?mSRG0s*vCyRdL#xniHMnkZ4-_n2DcQ*(+>W;A zMwB$fKM3fZoQ(#UZU20O1`Af`+e;&P-*|IsHc{!cf-z+_nCjbI|LVeJ3eP0ltaEbl znO`5y*McOpQVIR8%ofS$m^ch80cK3H1u7!m_29PTVDTpLD7SrKzz8?|lkZ(#UD0m0 zRQ%0Y=lKcM%ySL4-aiLvS&1AT+?|Sg5-X>#I5@_Ga^vX(iX#F%W@UuNIov34=gzG9YSsg)p zyht&USD2ivqd`Ccn=ZaKeDpX-kSPLVq3#hPA;Nf2)vjxPR3S{XGBzglk=GMra1d1) z9q8nyqs>R$a|a^l?pzTS85A-UhmrZ>+Kk}wGdceTb>gMJvVQ7^wI%0s8(Q4%W1L=&Z>)Dj`{4chbt6Q7-qLQjuX(im8*(pRf z3p|-g-c118B4U3>ewN1X=Rz;<8&@G+!~atf`f+D5Acd6F)n}{P^0std3IHd@4Ij)s zDz9F;v)vK3^1YU+2^ky zuPU#m>AEg2uY-jMio} zj*h9?+sm&)i6PPeRSW&T`n)bU`8A9rI#%r^#nnuchX^6j+OqqJTU1(P(j}49pe254 z-qH;b>1M(7(HwsDDe^5`_La9bCPSTq*dLRr%`q!F0Dj*hOw}r5e|h`S=Q>`h?39!@ zYw~cb@MWVj#(OFQwiogfN1;EXl2qvzC%D1PY)@^?5?(WuHGWWzHM3{1X6D?1*N)-1sl<|D`OpDwW7x`%Fc`iCrG+&mvMQhaGpZ;e(?ls4KzQ`CPu$XJXZ~ zNm8D-*9U_+R3x_G(NV>Vik#3ehFS{O;`<$($P(7%O%UmLUZNc`V$_#CDo9AF>+{9zu+TlJ zE&w*^9r%rgHiaJ@pB$J#-0~Hx&02JJ7h26AdT38nzz#dQX|crt=piEPp|kfsg5>vm z!oD+QihYe#e8Otb#S7$pHz8v(*hy?Poh;n9`SE;XG&+V3#?8r$-O43piaCr}QwiF*F=&Fi4MQiMjiDEoysR4F5JW)~6Fe~dd+RYSq zb?(P?t#-yWR*nr)kx{pT-l6l;pmT|&aT0*EwQ1Jwr=q*et;`tEqyDODSj=c$xZYk_ z)?lbr|FFO2y>i;2Tf{Pu3GM?}-cXuhwo+To@+}mh{N3yzIAW)*c7G@8`daa8bD#!M^>f}cdnS)hA0{@`g(Y%UsX*uVrP6y} zNADxzf45Mt#7UMBev+4R4nQiYN$RL#+1OMcfIak1wlXkkwqYSdB`45gkuTu!86~=} z@B}&#fVZ;)lQ~RQ20SZs=YdoJ>){Pwx>XbV*ojaEM*T`Sob*R~PlWevq`%+Nr+Ni(YSt`yV5sPX2Qge_bqIv7eS{UPNrm)=v2kI4XtD+)f0$e9h_YqtBwgagG!{U zYg6WZaI5C&Ey&!w(D;mb439YVYunT=J^%xfYGpBEqp5UXGU|rH#67JadwN{A_fDb< z@a}Z1ne~R|5A~=f91cf_C!TI`7vEc`1|Z27U%jQ_uM8flLAnm=T@;hto$ufISA;|d zlZDiqIqi`Rb9-~Y1F~oTwTun3*1iseDIon7=)2HjQflf z?@JVk&+#N62iqJ#)QFh8D@SPw`u#PlC;`b{lo>toXT&&Rn7p?q<{Du&jXB?xs>=kA zNl)-AX=BGb;{4kanUQ<;=V?PO8O5cAh#XV~7FxbFcM12-(;23wQ86_fBWrE>$w$s3 zByLRC9Vns({V43i+*R|uK3_U; za0yBedd>!H?13AiUv8UQVIVg5)}Qftc#tUql>+iFvTS*pfKj#OJwS zdg|Zan?()R^p;5%F6ZRxS{%#6OG*sBm-oh(y3Vz;2!stXGwZ9*--Q2--3`vz`sQqrGZh`1SNZ;MEFS zu>jvbpw1JP?&_7V_hBJ-=I~N)n*+L)ynY8W1R4F+DEhO$iJs2TxY8cs6YV8J?6{ct zO{7%^0}Bfala$JzN2Hk8u~ZtGFU$ZCOUAi4lASfL-rwy@54suCJKIc$?(7I+rjz)u ztl%pP5WYVU70QgQi5dk26(Sk4$i2>hnCE{cJBUku%tzyTNRJOIB?pbZJHku-u_H&m z(DX68gq!pXU5$0F;)%;UuI`eZN|#BsgNDm#&OvN+-vs!7b z*H@N+2LGn|HAy2<_u)lvc>SyCwpETwuBwczBELnyCw~?eFo+fOJ1cw?UW>3(1@;D# z2-2*VZmEmRW0d+>v>VU$hByuw9QN?ff04Eh=W2$Atx~5W zdWxw}o`upr`2IZ5l=spw&#d7(w3#}}%{o>8Q|(#MYx@JW#6+zSfg~wtK4N1_NiDx> z>2wEHME6o%BR%8knGwLPH@fKJ6#d6-f21q3`DnJi5!%jHLeT3oWO zoZ7)6rc`J^PSF6xxJ9P;5|F5rd(?}(hJL~nWZ@$!;Jh~_16`#7zz(4js42qD-pbRl zNe*BLYH7mxr~t^s_?l7TGh%ta(Vg(RRCvP#QGnC(*_FjhQvD+22}eyOWX@SHWf}8V z9>8N+C**c8^uovfNKtKmeb783p-f)NufVQe!L-89>MVTkX)_=jr#E4WnO@Rx(W{c@* z0usEMc@wOl5}7{>tBJaN%M5}h%q@2X;o;(3IA>$iZ)UtAAt7n+>469?C5G=eKGyz@9BGB$N?aFMW_d{rySch@ zug7-+w3z+XiN*BvbcSl4Lb!Wr^0No&*WyTCW0(L~J_%k^mHx{B7^*cHzz-wm2Vl0p z>0(|7e?+b4>+z^O558UeRon$E3<$6T&Ct;B8{nNapNz_iec`oR3<-2OMO2eZjRh+{%Ub zmR^xV(B{q%&Ea5Tn@$|iMOFB@&TVRhaHVZ(eJB~-v#Ht7A(DPO&6tGhq_YfQW z0W7G5lg#f}HIaY&3?EuM%xfaixDv@tuSePR@$o5MD&Xx|Ke$=V#}Tfa*Gu*_R z`u+QQFzlH?oDx{)IO7=Z7-~^11m#KiMiT|+w7Ys%{IXUs~ z9!wR`km>|1b%*Uq)_-rg)&QgM+04?7rqPkmNj{J~@7L>=ya|SXk%jaGz><}i176WRE8ta*No)^nOSgjg4*!{pLrFAOf zPVkP*L7M!H=Ow<2JQXaKi@le@yfRfZ5<{v_W}~2>Qa`*GfZgPTEphZ?+s#pEep{vs za&d8Sg%jAwm+o_?H4Yl7z*>+wK7Qpc6-Tbv64II#OaqE-%@z_3R}rymJ`9W^F8oWA zL~_1FEc^g@BIoMZ1p%`YVvBV9PFlE3qm>(sG0}(m^eGxYd?TOa4zLqD0q8;9VW-vW z8r&U3!m8Ekv@l*FcXn8S7Brk7v~(m$g$uZ- zeZk>9$*v*0ln9(%mtAsFSJA$^RhiEw;6N1J{p}r%w4G10G_727T}=y;)Zl!40*CBJ zL{f>c>31{Ai|$Wu>ba?@#k$tXGX*q)gBC;BwX3C?pYZYUsB@wVE)=3}0rp}Asjz4L z`yIfS#dX;lfX!}l!hS{}0ZWkVVY@D{u^e*ZKbi&2OKV12PnSOe01oGH`0;|v((>t) zK^eD@44*QY<@w%1TpBX~jYj_fY`u_iGU=$BghpOI#Ijx=YT}1302{H;Ok@4b?cR}b z^05=jkH=A3p)sKVf*nZB*nl^0ynT4PMKud> z`>a+%mzIeivhbsTy<493|9WwR+QW+$-WKBzN275QJXRAIcTq-ctlR(V6%qN50VGdd z^GzD{Uee(TZpEDjG literal 0 HcmV?d00001 From 18fa8b79b3485280bf61ff6411e3af1eed47f9e0 Mon Sep 17 00:00:00 2001 From: Miren Esnaola Date: Thu, 22 Dec 2022 12:19:01 +0100 Subject: [PATCH 2/6] Corrected load balancing scheme in backend service Corrected load balancing scheme in backend service --- modules/net-glb/backend-service.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/net-glb/backend-service.tf b/modules/net-glb/backend-service.tf index f8956e138..acadda3bd 100644 --- a/modules/net-glb/backend-service.tf +++ b/modules/net-glb/backend-service.tf @@ -60,7 +60,7 @@ resource "google_compute_backend_service" "default" { health_checks = length(each.value.health_checks) == 0 ? null : [ for k in each.value.health_checks : lookup(local.hc_ids, k, k) ] - load_balancing_scheme = "EXTERNAL" + load_balancing_scheme = var.use_classic_version ? "EXTERNAL" : "EXTERNAL_MANAGED" port_name = ( each.value.port_name == null ? lower(each.value.protocol == null ? var.protocol : each.value.protocol) From 082c63dfc5376abb420e77f50e8eb72c3fe84c59 Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Thu, 22 Dec 2022 12:27:09 +0100 Subject: [PATCH 3/6] refactor GCS module and tests (#1066) --- modules/gcs/README.md | 57 ++++++++--------- modules/gcs/main.tf | 44 +++++++------- modules/gcs/variables.tf | 84 +++++++++++++++++--------- tests/modules/gcs/common.tfvars | 13 ++++ tests/modules/gcs/fixture/main.tf | 29 --------- tests/modules/gcs/fixture/variables.tf | 77 ----------------------- tests/modules/gcs/iam.tfvars | 3 + tests/modules/gcs/iam.yaml | 30 +++++++++ tests/modules/gcs/prefix.tfvars | 1 + tests/modules/gcs/prefix.yaml | 44 ++++++++++++++ tests/modules/gcs/test_plan.py | 54 ----------------- tests/modules/gcs/tftest.yaml | 20 ++++++ 12 files changed, 215 insertions(+), 241 deletions(-) create mode 100644 tests/modules/gcs/common.tfvars delete mode 100644 tests/modules/gcs/fixture/main.tf delete mode 100644 tests/modules/gcs/fixture/variables.tf create mode 100644 tests/modules/gcs/iam.tfvars create mode 100644 tests/modules/gcs/iam.yaml create mode 100644 tests/modules/gcs/prefix.tfvars create mode 100644 tests/modules/gcs/prefix.yaml delete mode 100644 tests/modules/gcs/test_plan.py create mode 100644 tests/modules/gcs/tftest.yaml diff --git a/modules/gcs/README.md b/modules/gcs/README.md index acffd2e1d..439b4522d 100644 --- a/modules/gcs/README.md +++ b/modules/gcs/README.md @@ -1,4 +1,5 @@ # Google Cloud Storage Module + ## Example ```hcl @@ -41,12 +42,10 @@ module "bucket" { iam = { "roles/storage.admin" = ["group:storage@example.com"] } - retention_policy = { retention_period = 100 is_locked = true } - logging_config = { log_bucket = var.bucket log_object_prefix = null @@ -63,32 +62,26 @@ module "bucket" { project_id = "myproject" prefix = "test" name = "my-bucket" - iam = { "roles/storage.admin" = ["group:storage@example.com"] } - - lifecycle_rule = { - action = { - type = "SetStorageClass" - storage_class = "STANDARD" - } - condition = { - age = 30 - created_before = null - with_state = null - matches_storage_class = null - num_newer_versions = null - custom_time_before = null - days_since_custom_time = null - days_since_noncurrent_time = null - noncurrent_time_before = null + lifecycle_rules = { + lr-0 = { + action = { + type = "SetStorageClass" + storage_class = "STANDARD" + } + condition = { + age = 30 + } } } } # tftest modules=1 resources=2 ``` + ### Minimal example with GCS notifications + ```hcl module "bucket-gcs-notification" { source = "./fabric/modules/gcs" @@ -112,23 +105,23 @@ module "bucket-gcs-notification" { | name | description | type | required | default | |---|---|:---:|:---:|:---:| -| [name](variables.tf#L89) | Bucket name suffix. | string | ✓ | | -| [project_id](variables.tf#L117) | Bucket project id. | string | ✓ | | -| [cors](variables.tf#L17) | CORS configuration for the bucket. Defaults to null. | object({…}) | | null | +| [name](variables.tf#L116) | Bucket name suffix. | string | ✓ | | +| [project_id](variables.tf#L145) | Bucket project id. | string | ✓ | | +| [cors](variables.tf#L17) | CORS configuration for the bucket. Defaults to null. | object({…}) | | null | | [encryption_key](variables.tf#L28) | KMS key that will be used for encryption. | string | | null | | [force_destroy](variables.tf#L34) | Optional map to set force destroy keyed by name, defaults to false. | bool | | false | | [iam](variables.tf#L40) | IAM bindings in {ROLE => [MEMBERS]} format. | map(list(string)) | | {} | | [labels](variables.tf#L46) | Labels to be attached to all buckets. | map(string) | | {} | -| [lifecycle_rule](variables.tf#L52) | Bucket lifecycle rule. | object({…}) | | null | -| [location](variables.tf#L74) | Bucket location. | string | | "EU" | -| [logging_config](variables.tf#L80) | Bucket logging configuration. | object({…}) | | null | -| [notification_config](variables.tf#L94) | GCS Notification configuration. | object({…}) | | null | -| [prefix](variables.tf#L107) | Optional prefix used to generate the bucket name. | string | | null | -| [retention_policy](variables.tf#L122) | Bucket retention policy. | object({…}) | | null | -| [storage_class](variables.tf#L131) | Bucket storage class. | string | | "MULTI_REGIONAL" | -| [uniform_bucket_level_access](variables.tf#L141) | Allow using object ACLs (false) or not (true, this is the recommended behavior) , defaults to true (which is the recommended practice, but not the behavior of storage API). | bool | | true | -| [versioning](variables.tf#L147) | Enable versioning, defaults to false. | bool | | false | -| [website](variables.tf#L153) | Bucket website. | object({…}) | | null | +| [lifecycle_rules](variables.tf#L52) | Bucket lifecycle rule. | map(object({…})) | | {} | +| [location](variables.tf#L101) | Bucket location. | string | | "EU" | +| [logging_config](variables.tf#L107) | Bucket logging configuration. | object({…}) | | null | +| [notification_config](variables.tf#L121) | GCS Notification configuration. | object({…}) | | null | +| [prefix](variables.tf#L135) | Optional prefix used to generate the bucket name. | string | | null | +| [retention_policy](variables.tf#L150) | Bucket retention policy. | object({…}) | | null | +| [storage_class](variables.tf#L159) | Bucket storage class. | string | | "MULTI_REGIONAL" | +| [uniform_bucket_level_access](variables.tf#L169) | Allow using object ACLs (false) or not (true, this is the recommended behavior) , defaults to true (which is the recommended practice, but not the behavior of storage API). | bool | | true | +| [versioning](variables.tf#L175) | Enable versioning, defaults to false. | bool | | false | +| [website](variables.tf#L181) | Bucket website. | object({…}) | | null | ## Outputs diff --git a/modules/gcs/main.tf b/modules/gcs/main.tf index 960b23d2e..68b77077d 100644 --- a/modules/gcs/main.tf +++ b/modules/gcs/main.tf @@ -75,22 +75,25 @@ resource "google_storage_bucket" "bucket" { } dynamic "lifecycle_rule" { - for_each = var.lifecycle_rule == null ? [] : [""] + for_each = var.lifecycle_rules + iterator = rule content { action { - type = var.lifecycle_rule.action["type"] - storage_class = var.lifecycle_rule.action["storage_class"] + type = rule.value.action.type + storage_class = rule.value.action.storage_class } condition { - age = var.lifecycle_rule.condition["age"] - created_before = var.lifecycle_rule.condition["created_before"] - with_state = var.lifecycle_rule.condition["with_state"] - matches_storage_class = var.lifecycle_rule.condition["matches_storage_class"] - num_newer_versions = var.lifecycle_rule.condition["num_newer_versions"] - custom_time_before = var.lifecycle_rule.condition["custom_time_before"] - days_since_custom_time = var.lifecycle_rule.condition["days_since_custom_time"] - days_since_noncurrent_time = var.lifecycle_rule.condition["days_since_noncurrent_time"] - noncurrent_time_before = var.lifecycle_rule.condition["noncurrent_time_before"] + age = rule.value.condition.age + created_before = rule.value.condition.created_before + custom_time_before = rule.value.condition.custom_time_before + days_since_custom_time = rule.value.condition.days_since_custom_time + days_since_noncurrent_time = rule.value.condition.days_since_noncurrent_time + matches_prefix = rule.value.condition.matches_prefix + matches_storage_class = rule.value.condition.matches_storage_class + matches_suffix = rule.value.condition.matches_suffix + noncurrent_time_before = rule.value.condition.noncurrent_time_before + num_newer_versions = rule.value.condition.num_newer_versions + with_state = rule.value.condition.with_state } } } @@ -104,15 +107,14 @@ resource "google_storage_bucket_iam_binding" "bindings" { } resource "google_storage_notification" "notification" { - count = local.notification ? 1 : 0 - bucket = google_storage_bucket.bucket.name - payload_format = var.notification_config.payload_format - topic = google_pubsub_topic.topic[0].id - event_types = var.notification_config.event_types - custom_attributes = var.notification_config.custom_attributes - - depends_on = [google_pubsub_topic_iam_binding.binding] - + count = local.notification ? 1 : 0 + bucket = google_storage_bucket.bucket.name + payload_format = var.notification_config.payload_format + topic = google_pubsub_topic.topic[0].id + custom_attributes = var.notification_config.custom_attributes + event_types = var.notification_config.event_types + object_name_prefix = var.notification_config.object_name_prefix + depends_on = [google_pubsub_topic_iam_binding.binding] } resource "google_pubsub_topic_iam_binding" "binding" { count = local.notification ? 1 : 0 diff --git a/modules/gcs/variables.tf b/modules/gcs/variables.tf index 2e1517234..58aec4dfb 100644 --- a/modules/gcs/variables.tf +++ b/modules/gcs/variables.tf @@ -17,10 +17,10 @@ variable "cors" { description = "CORS configuration for the bucket. Defaults to null." type = object({ - origin = list(string) - method = list(string) - response_header = list(string) - max_age_seconds = number + origin = optional(list(string)) + method = optional(list(string)) + response_header = optional(list(string)) + max_age_seconds = optional(number) }) default = null } @@ -49,26 +49,53 @@ variable "labels" { default = {} } -variable "lifecycle_rule" { +variable "lifecycle_rules" { description = "Bucket lifecycle rule." - type = object({ + type = map(object({ action = object({ type = string - storage_class = string + storage_class = optional(string) }) condition = object({ - age = number - created_before = string - with_state = string - matches_storage_class = list(string) - num_newer_versions = string - custom_time_before = string - days_since_custom_time = string - days_since_noncurrent_time = string - noncurrent_time_before = string + age = optional(number) + created_before = optional(string) + custom_time_before = optional(string) + days_since_custom_time = optional(number) + days_since_noncurrent_time = optional(number) + matches_prefix = optional(list(string)) + matches_storage_class = optional(list(string)) # STANDARD, MULTI_REGIONAL, REGIONAL, NEARLINE, COLDLINE, ARCHIVE, DURABLE_REDUCED_AVAILABILITY + matches_suffix = optional(list(string)) + noncurrent_time_before = optional(string) + num_newer_versions = optional(number) + with_state = optional(string) # "LIVE", "ARCHIVED", "ANY" }) - }) - default = null + })) + default = {} + nullable = false + validation { + condition = alltrue([ + for k, v in var.lifecycle_rules : v.action != null && v.condition != null + ]) + error_message = "Lifecycle rules action and condition cannot be null." + } + validation { + condition = alltrue([ + for k, v in var.lifecycle_rules : contains( + ["Delete", "SetStorageClass", "AbortIncompleteMultipartUpload"], + v.action.type + ) + ]) + error_message = "Lifecycle rules action type has unsupported value." + } + validation { + condition = alltrue([ + for k, v in var.lifecycle_rules : + v.action.type != "SetStorageClass" + || + v.action.storage_class != null + ]) + error_message = "Lifecycle rules with action type SetStorageClass require a storage class." + } } variable "location" { @@ -81,7 +108,7 @@ variable "logging_config" { description = "Bucket logging configuration." type = object({ log_bucket = string - log_object_prefix = string + log_object_prefix = optional(string) }) default = null } @@ -94,12 +121,13 @@ variable "name" { variable "notification_config" { description = "GCS Notification configuration." type = object({ - enabled = bool - payload_format = string - topic_name = string - sa_email = string - event_types = list(string) - custom_attributes = map(string) + enabled = bool + payload_format = string + topic_name = string + sa_email = string + event_types = optional(list(string)) + custom_attributes = optional(map(string)) + object_name_prefix = optional(string) }) default = null } @@ -123,7 +151,7 @@ variable "retention_policy" { description = "Bucket retention policy." type = object({ retention_period = number - is_locked = bool + is_locked = optional(bool) }) default = null } @@ -153,8 +181,8 @@ variable "versioning" { variable "website" { description = "Bucket website." type = object({ - main_page_suffix = string - not_found_page = string + main_page_suffix = optional(string) + not_found_page = optional(string) }) default = null } diff --git a/tests/modules/gcs/common.tfvars b/tests/modules/gcs/common.tfvars new file mode 100644 index 000000000..5bab53b25 --- /dev/null +++ b/tests/modules/gcs/common.tfvars @@ -0,0 +1,13 @@ +force_destroy = true +labels = { environment = "test" } +logging_config = { + log_bucket = "foo" +} +name = "test" +project_id = "test-project" +retention_policy = { + retention_period = 5 + is_locked = false +} +storage_class = "MULTI_REGIONAL" +versioning = true diff --git a/tests/modules/gcs/fixture/main.tf b/tests/modules/gcs/fixture/main.tf deleted file mode 100644 index ea2e994f6..000000000 --- a/tests/modules/gcs/fixture/main.tf +++ /dev/null @@ -1,29 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -module "test" { - source = "../../../../modules/gcs" - project_id = "my-project" - uniform_bucket_level_access = var.uniform_bucket_level_access - force_destroy = var.force_destroy - iam = var.iam - labels = var.labels - logging_config = var.logging_config - name = "bucket-a" - prefix = var.prefix - retention_policy = var.retention_policy - versioning = var.versioning -} diff --git a/tests/modules/gcs/fixture/variables.tf b/tests/modules/gcs/fixture/variables.tf deleted file mode 100644 index 455d9a4bb..000000000 --- a/tests/modules/gcs/fixture/variables.tf +++ /dev/null @@ -1,77 +0,0 @@ -/** - * Copyright 2022 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -variable "uniform_bucket_level_access" { - type = bool - default = false -} - -variable "force_destroy" { - type = bool - default = true -} - -variable "iam" { - type = map(list(string)) - default = {} -} - -variable "labels" { - type = map(string) - default = { environment = "test" } -} - -variable "logging_config" { - type = object({ - log_bucket = string - log_object_prefix = string - }) - default = { - log_bucket = "foo" - log_object_prefix = null - } -} - -variable "prefix" { - type = string - default = null -} - -variable "project_id" { - type = string - default = "my-project" -} - -variable "retention_policy" { - type = object({ - retention_period = number - is_locked = bool - }) - default = { - retention_period = 5 - is_locked = false - } -} - -variable "storage_class" { - type = string - default = "MULTI_REGIONAL" -} - -variable "versioning" { - type = bool - default = true -} diff --git a/tests/modules/gcs/iam.tfvars b/tests/modules/gcs/iam.tfvars new file mode 100644 index 000000000..cfb3a0148 --- /dev/null +++ b/tests/modules/gcs/iam.tfvars @@ -0,0 +1,3 @@ +iam = { + "roles/storage.admin" = ["user:a@example.org"] +} diff --git a/tests/modules/gcs/iam.yaml b/tests/modules/gcs/iam.yaml new file mode 100644 index 000000000..8a85a4bd6 --- /dev/null +++ b/tests/modules/gcs/iam.yaml @@ -0,0 +1,30 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +values: + google_storage_bucket.bucket: + name: test + + google_storage_bucket_iam_binding.bindings["roles/storage.admin"]: + bucket: test + condition: [] + members: + - user:a@example.org + role: roles/storage.admin + +counts: + google_storage_bucket: 1 + google_storage_bucket_iam_binding: 1 + modules: 0 + resources: 2 diff --git a/tests/modules/gcs/prefix.tfvars b/tests/modules/gcs/prefix.tfvars new file mode 100644 index 000000000..0031d561d --- /dev/null +++ b/tests/modules/gcs/prefix.tfvars @@ -0,0 +1 @@ +prefix = "foo" diff --git a/tests/modules/gcs/prefix.yaml b/tests/modules/gcs/prefix.yaml new file mode 100644 index 000000000..6baee4a15 --- /dev/null +++ b/tests/modules/gcs/prefix.yaml @@ -0,0 +1,44 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +values: + google_storage_bucket.bucket: + force_destroy: true + labels: + environment: test + location: EU + logging: + - log_bucket: foo + name: foo-test + project: test-project + retention_policy: + - is_locked: false + retention_period: 5 + storage_class: MULTI_REGIONAL + uniform_bucket_level_access: true + versioning: + - enabled: true + +counts: + google_storage_bucket: 1 + modules: 0 + resources: 1 + +outputs: + bucket: __missing__ + id: foo-test + name: foo-test + notification: null + topic: null + url: __missing__ diff --git a/tests/modules/gcs/test_plan.py b/tests/modules/gcs/test_plan.py deleted file mode 100644 index 22775a589..000000000 --- a/tests/modules/gcs/test_plan.py +++ /dev/null @@ -1,54 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -def test_buckets(plan_runner): - "Test bucket resources." - _, resources = plan_runner() - assert len(resources) == 1 - r = resources[0] - assert r['type'] == 'google_storage_bucket' - assert r['values']['name'] == 'bucket-a' - assert r['values']['project'] == 'my-project' - - -def test_prefix(plan_runner): - "Test bucket name when prefix is set." - _, resources = plan_runner(prefix='foo') - assert resources[0]['values']['name'] == 'foo-bucket-a' - - -def test_config_values(plan_runner): - "Test that variables set the correct attributes on buckets." - variables = dict( - uniform_bucket_level_access='true', - force_destroy='true', - versioning='true' - ) - _, resources = plan_runner(**variables) - assert len(resources) == 1 - r = resources[0] - assert r['values']['uniform_bucket_level_access'] is True - assert r['values']['force_destroy'] is True - assert r['values']['versioning'] == [{'enabled': True}] - assert r['values']['logging'] == [{'log_bucket': 'foo'}] - assert r['values']['retention_policy'] == [ - {'is_locked': False, 'retention_period': 5} - ] - - -def test_iam(plan_runner): - "Test bucket resources with iam roles and members." - iam = '{ "roles/storage.admin" = ["user:a@b.com"] }' - _, resources = plan_runner(iam=iam) - assert len(resources) == 2 diff --git a/tests/modules/gcs/tftest.yaml b/tests/modules/gcs/tftest.yaml new file mode 100644 index 000000000..22337d18d --- /dev/null +++ b/tests/modules/gcs/tftest.yaml @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +module: modules/gcs +common_tfvars: + - common.tfvars +tests: + prefix: + iam: From 5063c3b513a2a461b1d5de504c76a75b6742c757 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aur=C3=A9lien=20Legrand?= Date: Thu, 22 Dec 2022 13:14:42 +0100 Subject: [PATCH 4/6] Network dashboard: PSA ranges support, starting with Cloud SQL (#1063) * Adding PSA ranges support and ip utilization, starting with Cloud SQL * yapf formatting * refactor * fix camelcase * remove unused variable * remove test file * rename discover plugin * fixing issue with asset name Co-authored-by: Ludo --- ...iscover-cai-compute.py => discover-cai.py} | 66 ++++++++++++----- .../src/plugins/series-psa.py | 71 +++++++++++++++++++ 2 files changed, 119 insertions(+), 18 deletions(-) rename blueprints/cloud-operations/network-dashboard/src/plugins/{discover-cai-compute.py => discover-cai.py} (78%) create mode 100644 blueprints/cloud-operations/network-dashboard/src/plugins/series-psa.py diff --git a/blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai-compute.py b/blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai.py similarity index 78% rename from blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai-compute.py rename to blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai.py index 86379fb1c..1d22cd8e9 100644 --- a/blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai-compute.py +++ b/blueprints/cloud-operations/network-dashboard/src/plugins/discover-cai.py @@ -24,21 +24,22 @@ import logging from . import HTTPRequest, Level, Resource, register_init, register_discovery from .utils import parse_cai_results - CAI_URL = ('https://content-cloudasset.googleapis.com/v1' '/{root}/assets' '?contentType=RESOURCE&{asset_types}&pageSize=500') LOGGER = logging.getLogger('net-dash.discovery.cai-compute') TYPES = { - 'addresses': 'Address', - 'firewall_policies': 'FirewallPolicy', - 'firewall_rules': 'Firewall', - 'forwarding_rules': 'ForwardingRule', - 'instances': 'Instance', - 'networks': 'Network', - 'subnetworks': 'Subnetwork', - 'routers': 'Router', - 'routes': 'Route', + 'addresses': 'compute.googleapis.com/Address', + 'global_addresses': 'compute.googleapis.com/GlobalAddress', + 'firewall_policies': 'compute.googleapis.com/FirewallPolicy', + 'firewall_rules': 'compute.googleapis.com/Firewall', + 'forwarding_rules': 'compute.googleapis.com/ForwardingRule', + 'instances': 'compute.googleapis.com/Instance', + 'networks': 'compute.googleapis.com/Network', + 'subnetworks': 'compute.googleapis.com/Subnetwork', + 'routers': 'compute.googleapis.com/Router', + 'routes': 'compute.googleapis.com/Route', + 'sql_instances': 'sqladmin.googleapis.com/Instance' } NAMES = {v: k for k, v in TYPES.items()} @@ -61,7 +62,8 @@ def _handle_discovery(resources, response, data): 'Processes the asset API response and returns parsed resources or next URL.' LOGGER.info('discovery handle request') for result in parse_cai_results(data, 'cai-compute', method='list'): - resource = _handle_resource(resources, result['resource']) + resource = _handle_resource( + resources, result['assetType'], result['resource']) if not resource: continue yield resource @@ -72,15 +74,18 @@ def _handle_discovery(resources, response, data): yield HTTPRequest(f'{url}&pageToken={page_token}', {}, None) -def _handle_resource(resources, data): +def _handle_resource(resources, asset_type, data): 'Parses and returns a single resource. Calls resource-level handler.' - attrs = data['data'] # general attributes shared by all resource types - resource_name = NAMES[data['discoveryName']] + attrs = data['data'] + # we use the asset type as the discovery name sometimes does not match + # e.g. assetType = GlobalAddress but discoveryName = Address + resource_name = NAMES[asset_type] resource = { - 'id': attrs['id'], + 'id': attrs.get('id'), 'name': attrs['name'], - 'self_link': _self_link(attrs['selfLink']) + 'self_link': _self_link(attrs['selfLink']), + 'assetType': asset_type } # derive parent type and id and skip if parent is not within scope parent_data = _get_parent(data['parent'], resources) @@ -145,6 +150,19 @@ def _handle_forwarding_rules(resource, data): } +def _handle_global_addresses(resource, data): + 'Handles GlobalAddress type resource data (ex: PSA ranges).' + network = data.get('network') + return { + 'address': data['address'], + 'prefixLength': data.get('prefixLength') or None, + 'internal': data.get('addressType') == 'INTERNAL', + 'purpose': data.get('purpose', ''), + 'status': data.get('status', ''), + 'network': None if not network else _self_link(network), + } + + def _handle_instances(resource, data): 'Handles instance type resource data.' if data['status'] != 'RUNNING': @@ -184,6 +202,18 @@ def _handle_routes(resource, data): return {'next_hop_type': hop[0], 'network': _self_link(data['network'])} +def _handle_sql_instances(resource, data): + 'Handles cloud sql instance type resource data.' + return { + 'name': data['name'], + 'self_link': _self_link(data['selfLink']), + 'ipAddresses': [ + i['ipAddress'] for i in data['ipAddresses'] if i['type'] == 'PRIVATE' + ], + 'region': data['region'], + 'availabilityType': data['settings']['availabilityType'], + } + def _handle_subnetworks(resource, data): 'Handles subnetwork type resource data.' secondary_ranges = [{ @@ -201,14 +231,14 @@ def _handle_subnetworks(resource, data): def _self_link(s): 'Removes initial part from self links.' - return s.removeprefix('https://www.googleapis.com/compute/v1/') + return '/'.join(s.split('/')[5:]) def _url(resources): 'Returns discovery URL' discovery_root = resources['config:discovery_root'] asset_types = '&'.join( - f'assetTypes=compute.googleapis.com/{t}' for t in TYPES.values()) + f'assetTypes={t}' for t in TYPES.values()) return CAI_URL.format(root=discovery_root, asset_types=asset_types) diff --git a/blueprints/cloud-operations/network-dashboard/src/plugins/series-psa.py b/blueprints/cloud-operations/network-dashboard/src/plugins/series-psa.py new file mode 100644 index 000000000..0ea088f9c --- /dev/null +++ b/blueprints/cloud-operations/network-dashboard/src/plugins/series-psa.py @@ -0,0 +1,71 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +'Prepares descriptors and timeseries for subnetwork-level metrics.' + +import collections +import ipaddress +import itertools +import logging + +from . import MetricDescriptor, TimeSeries, register_timeseries + +DESCRIPTOR_ATTRS = { + 'addresses_available': 'Address limit per psa range', + 'addresses_used': 'Addresses used per psa range', + 'addresses_used_ratio': 'Addresses used ratio per psa range' +} +LOGGER = logging.getLogger('net-dash.timeseries.psa') + + +def _sql_addresses(sql_instances): + 'Returns counts of Cloud SQL instances per PSA range.' + for v in sql_instances.values(): + if not v['ipAddresses']: + continue + # 1 IP for the instance + 1 IP for the ILB + 1 IP if HA + yield v['ipAddresses'][0], 2 if v['availabilityType'] != 'REGIONAL' else 3 + + +@register_timeseries +def timeseries(resources): + 'Returns used/available/ratio timeseries for addresses by PSA ranges.' + LOGGER.info('timeseries') + for dtype, name in DESCRIPTOR_ATTRS.items(): + yield MetricDescriptor(f'network/psa/{dtype}', name, + ('project', 'network', 'subnetwork'), + dtype.endswith('ratio')) + psa_nets = { + k: ipaddress.ip_network('{}/{}'.format(v['address'], v['prefixLength'])) + for k, v in resources['global_addresses'].items() if v['prefixLength'] + } + psa_counts = {} + for address, ip_count in _sql_addresses(resources.get('sql_instances', {})): + ip_address = ipaddress.ip_address(address) + for k, v in psa_nets.items(): + if ip_address in v: + psa_counts[k] = psa_counts.get(k, 0) + ip_count + break + + for k, v in psa_counts.items(): + max_ips = psa_nets[k].num_addresses - 4 + psa_range = resources['global_addresses'][k] + labels = { + 'network': psa_range['network'], + 'project': psa_range['project_id'], + 'psa_range': psa_range['name'] + } + yield TimeSeries('network/psa/addresses_available', max_ips, labels) + yield TimeSeries('network/psa/addresses_used', v, labels) + yield TimeSeries('network/psa/addresses_used_ratio', + 0 if v == 0 else v / max_ips, labels) From 2affc49d0052e83c857bbe7efa660f9c9039ebb8 Mon Sep 17 00:00:00 2001 From: Ludo Date: Thu, 22 Dec 2022 13:32:19 +0100 Subject: [PATCH 5/6] update changelog --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bd07a2469..6d2a7aded 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file. ### BLUEPRINTS +- [[#1063](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1063)] Network dashboard: PSA ranges support, starting with Cloud SQL ([aurelienlegrand](https://github.com/aurelienlegrand)) - [[#1062](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1062)] Fixes for GKE ([wiktorn](https://github.com/wiktorn)) - [[#1060](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1060)] Update src/README.md for Network Dashboard ([aurelienlegrand](https://github.com/aurelienlegrand)) - [[#1020](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1020)] Networking dashboard and discovery tool refactor ([ludoo](https://github.com/ludoo)) @@ -19,6 +20,8 @@ All notable changes to this project will be documented in this file. ### MODULES +- [[#1067](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1067)] Corrected load balancing scheme in backend service ([apichick](https://github.com/apichick)) +- [[#1066](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1066)] Refactor GCS module and tests for Terraform 1.3 ([ludoo](https://github.com/ludoo)) - [[#1062](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1062)] Fixes for GKE ([wiktorn](https://github.com/wiktorn)) - [[#1061](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1061)] **incompatible change:** Allow using dynamically generated address in LB modules NEGs ([ludoo](https://github.com/ludoo)) - [[#1059](https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/pull/1059)] Read ranges from correct fields in firewall factory ([juliocc](https://github.com/juliocc)) From e1613ee03eb36c7d79a0c2e3ee1dc1be5cbf367c Mon Sep 17 00:00:00 2001 From: Miren Esnaola Date: Thu, 22 Dec 2022 15:30:24 +0100 Subject: [PATCH 6/6] Added endpoint_attachment_hosts output to apigee module --- modules/apigee/README.md | 15 ++++++++------- modules/apigee/outputs.tf | 5 +++++ 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/modules/apigee/README.md b/modules/apigee/README.md index 7754d1023..fb26c1e72 100644 --- a/modules/apigee/README.md +++ b/modules/apigee/README.md @@ -180,12 +180,13 @@ module "apigee" { | name | description | sensitive | |---|---|:---:| -| [envgroups](outputs.tf#L17) | Environment groups. | | -| [environments](outputs.tf#L22) | Environment. | | -| [instances](outputs.tf#L27) | Instances. | | -| [org_id](outputs.tf#L32) | Organization ID. | | -| [org_name](outputs.tf#L37) | Organization name. | | -| [organization](outputs.tf#L42) | Organization. | | -| [service_attachments](outputs.tf#L47) | Service attachments. | | +| [endpoint_attachment_hosts](outputs.tf#L17) | Endpoint hosts. | | +| [envgroups](outputs.tf#L22) | Environment groups. | | +| [environments](outputs.tf#L27) | Environment. | | +| [instances](outputs.tf#L32) | Instances. | | +| [org_id](outputs.tf#L37) | Organization ID. | | +| [org_name](outputs.tf#L42) | Organization name. | | +| [organization](outputs.tf#L47) | Organization. | | +| [service_attachments](outputs.tf#L52) | Service attachments. | | diff --git a/modules/apigee/outputs.tf b/modules/apigee/outputs.tf index a5e703881..74ad9f18d 100644 --- a/modules/apigee/outputs.tf +++ b/modules/apigee/outputs.tf @@ -14,6 +14,11 @@ * limitations under the License. */ +output "endpoint_attachment_hosts" { + description = "Endpoint hosts." + value = { for k, v in google_apigee_endpoint_attachment.endpoint_attachments : k => v.host } +} + output "envgroups" { description = "Environment groups." value = try(google_apigee_envgroup.envgroups, null)