From 444666faa47186d30d222eb0ebe4a25f1649f6e5 Mon Sep 17 00:00:00 2001 From: Mark Morton Date: Tue, 16 Jun 2020 14:14:50 -0700 Subject: [PATCH] New source files and Makefile update for Test Manual (From yocto-docs rev: d7cff640569a5772f3c366b4136762628fca534d) Signed-off-by: Mark Morton Signed-off-by: Richard Purdie --- documentation/Makefile | 10 + .../test-manual/figures/ab-test-cluster.png | Bin 0 -> 18684 bytes .../test-manual/figures/test-manual-title.png | Bin 0 -> 15382 bytes .../test-manual/test-manual-customization.xsl | 27 + .../test-manual/test-manual-intro.xml | 634 +++++++++++ .../test-manual/test-manual-style.css | 989 ++++++++++++++++++ .../test-manual/test-manual-test-process.xml | 109 ++ .../test-manual-understand-autobuilder.xml | 312 ++++++ documentation/test-manual/test-manual.xml | 103 ++ 9 files changed, 2184 insertions(+) create mode 100644 documentation/test-manual/figures/ab-test-cluster.png create mode 100644 documentation/test-manual/figures/test-manual-title.png create mode 100644 documentation/test-manual/test-manual-customization.xsl create mode 100644 documentation/test-manual/test-manual-intro.xml create mode 100644 documentation/test-manual/test-manual-style.css create mode 100644 documentation/test-manual/test-manual-test-process.xml create mode 100644 documentation/test-manual/test-manual-understand-autobuilder.xml create mode 100755 documentation/test-manual/test-manual.xml diff --git a/documentation/Makefile b/documentation/Makefile index 2f7292214f..6a369637f5 100644 --- a/documentation/Makefile +++ b/documentation/Makefile @@ -358,6 +358,16 @@ STYLESHEET = $(DOC)/*.css endif +ifeq ($(DOC),test-manual) +XSLTOPTS = --xinclude +ALLPREQ = html tarball +TARFILES = test-manual.html test-manual-style.css \ + figures/test-manual-title.png figures/ab-test-cluster.png +MANUALS = $(DOC)/$(DOC).html +FIGURES = figures +STYLESHEET = $(DOC)/*.css +endif + ## # These URI should be rewritten by your distribution's xml catalog to # match your locally installed XSL stylesheets. diff --git a/documentation/test-manual/figures/ab-test-cluster.png b/documentation/test-manual/figures/ab-test-cluster.png new file mode 100644 index 0000000000000000000000000000000000000000..6a6a7882b43895cd3e81e18b8829c88af5b2ca8c GIT binary patch literal 18684 zcmcG#1z6i(x9^F&OKEYpmH=&WcL?q+PLUROcWBXKArM@GLxED0sJ{AVzKPGcX zdx$q=H!THeq{?CH9mEH;k5Z~qNJ!Q3xOe90h|k#0itpW!kO+P~y^xb%5>O!_DJCk( zO6ho;94z40lKsw-IA&yftZ8{Kknx!u4Oy;+wME_nOdqQFHfDtHdABXmZ!~OuDV>Cp zNJ=P*EtwoyNUkO$^iLzn8m>Mzjx6!$^P$bcQ;@#&;O2gx()_~wqrcs*09?SM66 zhhQc~C_Qf{qQr#f%aM`nig?b5N|-SO85wa%Q2(>!f*DBUU6J#$vJeyBQlahKuz9^p zxgpk0Bn9-Nj>q=2w>F>`5`nD)3{btgPk$L}%(_sRp$SiE2Jr>vm#4nI?+6Ug40V{RPa}V+H+^B?eSZO4pwxVa=;~{TO9M`7j}H84%XVA*q5z(aY8Jya$i;ORK#(VZ#j@muj$ox+vPJ5XgUlJmZRsA_{U{lN7o&vVq9=0qP zn2hZulXwu=-%1hO?U5FCA+k#a++KRvooF;CjxDbKj&bNy0|u1*nQ*WT!@&P>>q2cN z0d$EMyG+F|3Jy5i9IDwvk$P4o6PUjD`!9=5tI3$A0*zHzCivBtc5zJu#LT^@_1M#E zc;*B@T=0xqcAhqmnDIbEvk%*ka2pot0c{8tOBpFdYO4DwGa0)-`4mEZ+numFbrz8<3(>YhvWRedD^W*f6;QjRE|)MGMUf zqOKoz?-#t7&|Wa`scSTu5JEb7X`{gk3?~5=*lssbDrPJ)8u*2{>hT`w4ylh*(Y&db3Niju->Q?-WnFOP;q=bEr0G z8C69!!FC0;>5Rupgt(kNLrbrmU9(VX#%p+@bkphK7_%XnR$^i9I(4F0A}g5I77`R>8hMhXeJA;D~n#W%xeW6Io3)V64d9a|QY;nY5J zezdeS@6tm9U^_zWj)v=LQ!cbV1-gI<2VLv^>FQ0dj$32RpQvM#XifL+F1g<{UI4@#w!WoTI0Am_LKqVwww? zfj2vwjcai)!K(%5;r=W7gz^Dtx4Kb+Bg6{S=3%3aE^E<1!yCZ@V5rQ|V%vf1)^L_* z)LAat7b}pRd6Yv(Gzt7$JiO2+M~4F(umZA(vAHa-#huIbj$;pUhABdcJLd)g z_4bL4)L1HkXTT7wlo$$b{&>;TNHNyFYtVFiK#TEV&4{=s_M4BxwXn)_1Hp%Bi+C>T zczCdA;%0!>Ys4bd(f0<5WV>*=95!t|s~_89 zSmv#m(6PeGXR^{t14n4%#p-koDGvsq2>Plj)M*H!39ekE%dE5#)1vj>?Q^@qqWk5^ z+yOrt-rxUnZIF|#qjxa8$Sr@tG6{|NNcJeK*!)><1U?h}E|duxo)AZUqij57l~Ws{ zgk20~-YQS7g@IZd%f{XwC4nr0^9QNkSbq1}t{g6OS~sWTW7Fxm9_cfu@zK(au$8Et zQ_U3o>xip!lo!EZKUY5lNXo3z8#7UkgR#htPLkilEOI^gC2t% z-)&uP?gV`MVNc*zm1(U9*!nn`%x^!fF@X{8v2L?16uC^_L!alxAZ(Lcp#qmzacwtH z^5fc*>%4#6mrza=DWwYdE@aGPz! zb{t3zqUGt`p9fW$J}w*@dzRjh6Qert?9F&dw1{1&wCt0JpQIssa&*P-?wQ`~+bzCp z7zVJI8|Hp4g7*dLcvPBRFuieGMRWVT{rB?Se$pBXq4T%Ewt9tSB%LRlWmK7g+bzfkAin7`x^g7<*X!g8rXZ(PlKA1ju z;c~l!;?jcV*2$=T@O;rp;?4c=^=5dX^1;b*R)*u-CF7R*gos@CU*FB;wiL1({U_Kn z*lyKn-W0J5;xw_wb@6qGxfKOe1TLs1al0sJzv+r(^QNY+6pL#nC}$UAw_)(!F)LBF z3?9qc7unw);T=UBpCyC zm-eU4AzOHdMsggtMBC@$;OF_2mI< z4cDT(9Iz{`zC0v<6v}mHnG8k^52$2UWMQd-?MxFh*^{$}7FR-LFPM_U|Eju=NELij z!4Cu6HRd#K!9bED3Ol=1eUkl*%10t(c+Rx7)p@Xj+G;UZn8Rjz3uGo>TE3595~9Tw z_164Kv!0T)s&IC%Ehy-0cbCtx9QAaRcGJ(T+qWUb$t$cRH^P$-jiI)U#Ya?fm>71w zwpF#M{Ma0QdI5R)_`x3iO3Sp#)k^l z5-NM7ID@f(!M3xlEk7OLOLIq?0$!YoqD@Zf0HiYybg>-eF`%~(w#nR$2(Q}UI-H-q2xwa>B4wj);>bn`4Q zLaDP~mKUwd{vx5aqD8^=eGL^;E+hN9M^so$LH`JHwCFlsCf1dgb|vKf)ZJ4rxYCq0*e!bC~Wf!Dt(d& zi}^TIKZo@p7j83MWm+}Dy(KA4UefLq0^2+5=M{cu(GjTni8&ZgN1{cGNTYwKEj&@m zQbPV-l8;&5gGS+=jY}csO!Jb;#VSi zPchC^H>J@J;*|S3dxY6syKW@xvU#wNE!=4yY`KuNq+a)&j%2H|>23xsvgB>8N1i1) zCNfr*jIG#62HvNyw3!_@lU4R6V{VWP7#B|E*-u>V#tMR78AIAh^QAflh`Mb_mieJ=IS+o(} z57%2Uz8k;tE2wM7!~>DK~pthmu4J%%j0Z?l1jj_R!nGvuaOSSxMm_# z*M$UG8=<8qO$U+}nWA&_AZ)%&?>n=a?Tm5z_D=rOQt{*YH1}?WpKoluO#n6tNr0PJ z+lDpf^4|v*pt0e$%?=DHbRQDQe&g;XwqsxW+d?5Up@7zoCAO`a9j$ntNa(k+uCZNqfIe4f3z`9tca1`e>C zfUCe3)r+$$5Q|(P6Wa5yhNTg0WuL*@?zz97L&VYe^4Rpz5xZB95Q}S1lSQ7ftRjWb zS<7I2Kb^#JbQn9EHnvwdVJ5- z7gUsILqlmjViTDZo;5G*efDPD5u(_7 zal$xm^hpr=Z<^cRZ;T17lt5!iXaE}Kd=q_J*U1vMs~!!H^H)fqY{(=G!qA!AKN02pSmbZPRuNlg+ugFHL(c!We9^>oE_FMR+1ixrH}bJmgT}v|p-~oXSSGE9kiU{AM&hH%%>CN#C@55!P2j{^wt@ucRx& zvQ$b+ud`GVe2`ea!0&Ofu{g4uk0 zY1RdmTxmI8?20NiTe6-9G7sfgCYP~|PZgwr6wIc%UdZEO&lQ?=P{+Xfn9ITVc+SKH zIG~X#V!oWTCKBG+&9S&Mc9t4${}o(MY%GJqU#dTVGl*-ntg130=oPm1==R@-r0DcJ zozQ1|#kr{S7BO2CzT&$&F+R(YB_~i{O+Au%c9kv+K{NsP`<%!9cn6RCg1&}%D^@&- zei}L13KgIO5nr{f{*a*B1+jf_H_!Vmm9{c%k#zTKHt1AXAMirdtyw!#bm=mT?yeJ% zXGBG9W%wPLAV0o0HDSZ7o!|INskz~1L^nIL(x$(%n>gdHu(!Ng!<;!(3qMPqXkztJ z+YOt)Gq!b^L?j&xC5kTLEn2=h;S0TzpR%Rum@hj=jZ1ca$;@sba{Gxhmf-hZ}&y3bHdHT5}Vn+6yVXgrdj6LWhlO^UwG)%uq8&yFKzIKAVL}V!nQfB%Gy) zbPn&5INyEFOB=%B_}}tYyR7|za;ki_E1lafX+oo0ev8I{CN#>zi5)C_y{2gU-x}JF zQ5p`eHSfk~h)CB|MiFKN%VVE*sp~RM9;T8h1%;LW8WEjomBq2=#ilj21JGZ*Y^>o(1gLn!RJXyY5 zeo49HY){P_L139Wa(i+X6rhgb^G(Y$2KF5bB6{@t{MnJ!P5-c`zRT84jyXJW8ohQzZOHjqKb-&fR&$L9LrAvZ+qZF=Y2_o*xG{2rI&-5FN$b$#&3^0oLE)UE&s&R z9l4IZuldnb(rR znbtI^z?oX4`1;5);x|izQ_rRh9w|gVUwezdF4p%$bI=-!$2DcrWH&)({M#7yN zBvz){7>%!%gM4kt?~IGB$;p?eG5$Zov41rkVSD`7SlYiOFK|3KIB=Xyt*@`Y{DMo- zT;Xm#n7T6cv*sk|a-cA9^Z2oW1;XEA1U)|X%A%|odk2a!Wu@o{-_+CJa&skGW{#zk z5N{~KQ~Ukv=|@Mk3>;&su%*RNRE>t{RGQ`hiO~>>QF;2={#?^}T-xi_AOOv>Yw3&6 zS2HK@>ZX&kawDB!3pUdKTLAcPjsCCj!vAt|4?8C(ukt^&2%^bt)Hv!;2=Qw(%-7T0 zIl>LnkJdeVRLl6E+nzxfBw;Y}l>?Sxk~2N$P#Ce4HvgEP*EBhKok@w&ND)}y^D4`) zaK~q#3?fru2J06p?a0(`KQTO8Zr!;v*5tevlE>!gRHX`vorCJ}`2DThOnKjs-qsw{ zyHP+&zKk%W{-2;yGM(U-HPG6SN4~gC{lkZ_0p87!nKgUbOYD948RwTX`nCqY@_{_t z6Ssm_I}*9^s#|K52d=`s?bh`0=8Xj|0Gq!QdLt?U%?F5K&~=&dPs6G$I@=&8{ho-M z2r-d^Qwy=%85S#C359+rMPO_Hw;+eCxiUPHczW#vANI6dlzD~AMHnFHF~jR%?9t8$ zh?Wu5z6gLnPSCd$*sp9UzUTA!W7(jWhF|410o1rhCttpyJ%7=1cVBr&v3hCR;Rwl* zuq!j}H@*8|8b_hNcs3-hOHNYu`)wbjj5kO^Q8#^YD~Dn(eH|j~^zx~^8q=Kn!5OBW z)S)^4j2``0UQK^1+gJLfJ?Kg5SKNy6c?IUh$15M_8pj=y*-o9mO48PQc(B*IWFC(k z>1P4k!O!x`^cZ#x@~~fa2CQPWfOo$=&en7THgGi_Rv)v#t!h_htPpH< zE6R4s=j3n8yg_wcJa>In7sbi@@n>_bC1)Sz9$h~-p4C$Z#)0KbMYjA~TCE;bJzt=A ztv)EEG**b(D<_;Cz`TN5z@EH1VY`Lv11_3dJ=MZU(LP=y2>X!B$Q2T>Ku)1503R~k zb?qkA2@Z>IzI?7@=is}>(h|q5vcAS0xSmggIY)Q!7k1cVI)7?)ll($?XmTLX5s-QN zZQ)P#bX7pXM^nGA{@2LB7b9ZMHA2Uu`9sUvXF*MC2CgR5+}J(K1AnW0V}zZ`R}teE z3Nq{nxFrX=U=fLLcT|4?o zeZ^C&Qn<{7k0JTfircG9wSs6(mMHO{l(Nc#atq)wOaw!Z*^Wz13;WYqRiz1{)bak3 z+4~A+l+M$VEpfRq4hi=kenk~G)qS8a7WxZN$o3644u92%)Ud>cIn7d5cdRe+kdCnv zNiTM`-U!_0IjiSHOr@^Y&**Z`uS{LQB+0tVo}OFuIT-n&1MIdENbBb+PrM}7!^n|N|u(;Y6JtPBW0FhXo! z8FLlJ$GVOAEb%Dd(OJO?jpYx~-6vC+OHuNNuY|PplV0)L>xQ?aIi4;n3N6j(d znB!4wxR+aw>p@<~jK;F_D%1o##*21f_en;E()%irpHR6k$x|4-x_;*^HY{ZS%d92# z?rouHsjXZ>O57_396)D?|BDd+(mo2-?Ryj&lyz>ekb_zp^n#bSWAC?StR{SrgRa@` ze>&(Qsw-C$3zW_4jtnQD(Gnih2$eyKwuOWE1roG!0rldBop8u!EGqOEFeBZ9h6E?rglI4`nmL&42jYkv`)RF$U*lZw1dp;~1EV7nfJ1p4nKsSOdi=q;d2qQ68#X_x zmu0CYX%UpIz;uH>OA5p(K(_!Afiz``x`$%FCqi}{Oz7IoYuRi;V3}@lMrgsP3^B5l zb=x!0k=Ko+0)8n5`0yq$+x@xJPw_TWOdjf&*AMMlJ$s@}>RD)Ux&`T%-k&`wR2kHvy;3;44`2ub=P+pXqgQ{ZI$ z;9wN#AbGyRoJF;I97MFU`sQLbjw>VP^7zr^d`HVq)PJjpYRC&#$+gbF zPQz>e?(cOX8-R|lf3xpFyXPwhPMmaQl!fz<(@3U`X}PHv;{HAVS5|i2t`8{!i78Hq zWeJ_)2^f3LcpPY>V$KGSm7BU8vAXone5AO{ldTIlvv&J)eVvH0Jc!FWEz04*^}c3m zGfF0&+Oposx5vSwQ>@=d;ySH+|MZ8UQ;qWXs(~y>WF)lCGI;NAOTK{yCIIaX$6qBr zGoc%!j+sO{$F0n+ta5fMI+U!%2oejj0F%D~IoXb}k(6C@uDD9^Q|dEgAY{iQlWR>> z=zj`;iLK}`xZ)^Xaeb!Qj_CzaavmiGJ=*9|jE?SaGxYVnR74mOA})EBGN`6n>ik=% zWYNWH%Nelp4!HW(kX_>%hv_7OF8;yURULOALni2}B=hFu<2q3hRzl$3N|oS@jVBuM z@^Jgrhq-P|Oonq|*=$EbQuXY^QNWK}KUPM*zmcaaYy`^Q9l;k&17BIJP=MT?jqZWhUX zg;B7-5{crMps%g8N@rC|IJ)IhXo6_Z&MjsAQgk?=Fb1HhW|mC5@63i=PhK%|uH~0q znVE>smv42LRw>!>SG^Zrjzy?3Z@*=%dZNVfx1BzIE_DFNvsI+@(5GMQ_0x_;MK&pDv+j^4iAhgHG0Aa3+|I66xWoqHEP8_#LJRhxE% zNA3>LZ4n~_fikHinCIT0lJ;MA8c@)#>Uv%++Z!A+EuD@E0t=JX)QI-`d+8kwlI<5w zCk8VqW!gcR%HZSmn;RpnX)M>kC?006nZD1FB_zI?3StNN*1s=NlSh-%`k4z$hQTCx z`xH!SqC*wB{F-GFQ8rCc8W#_v2wx_#t|SH^4p%36w;C?)`FN6$L9*L!K=ngexy(kk zM|8OXz7&FvS%?Ci_dL-Fg zZ+uv8SrvtVrY%TGCBKre@u(4rDs)-dc{_!>jVy80iY}g%8Q*m*tXgw2Ju@thU4)3e zz*ZeZS8&&-_C02U)6S0wPH zoO(m3v4q61+-8QqG2mu%uc5072riRn*X82qQ34?+2i&BVjHNEdGwWu0=g#LKGAYJz zMlm#Av;5MVa789S#*ne-dn+_~jH&i#-dUWQ>DJ|&gdb99YNh;dVR?0A(PJ7uOd8_E zO?|P~jT`86%e1CH3QF8RBZZl_otMxjCy-8)-OXYvz1H zRMA_7-WJIkO$?8ss4sw}Er+Pio-IiW1kPr5-l2bS(X8b*BA3-I5gz#i6B!-()0huM zOJ?g!K>Fg2)E@Aa-+7=P;Jz2B*8-)z+$r_h^;hacLO7}ZF0QdMMq>u6=?4)fMGyL^ z9KTl*Rs>fxQqW#Em)AMevL3IaGW^t18ZWzXpG`Y-3ZI_4c@=KvGC?ch6K?9VanfUc zIg9m{dIxXcKF7B@cv-%nXY)EKv-%WD;Jj^cdZwu1!{Yj$HpCsBw;F`%S6c5Jp$YwTvP3kKF z+8mKmkd2L44jVHd(|HLcgC87uP7KN>^0MB4R%2xM$>Ss`>`IuZ%rh*UWKox1A ze3DG)(mOswQq+o$*|UU}i(P0KZ#y%FiJ_LU@+L_&%^(36K5jo9{ z9E9ULS2a{f;k#w^_&f+VpR6o7yFgbdcy*UA@6-HpU#hBPyrouo)Vqu)J5v+*iRvXq zV*wfDvKkpYtU;Nym^hmnmA`-IWK-1G6|igLWq|!pIwe0=C>i8gC`B3LUlA>R2}XRH zx8FYDWHSRVuJDYX)LIR;Du#xJw+K1bf}Swk|{Dc%2+-+-f#V8w9LRLa!INa9f3ghi0w+OZqqeeWxBiR`cWYgA`Zu^p= z=xedOIk{Dg1-TXf_vjz*#yEjGcAA$9qAt-hXA*u*6`$DH!QusJhP-Sik+K0-WVChUG7J^D-A)<`w!p|xwl>~y zbuCP5Ipmcvx`}KGx<~>PyJCSN2!wx9!@ndMsqypuKvVerE_APx#ONKRZm8W168w%Z z`!!6xb?}w(8F**1Dt<_2=b%i)sZ2?u`ELJg2>at#v*4KO-61i@ifZ4W{i+26i+hWz z4iCFY^WgkfAIPmJWR|b@zcXy+p~Gh#z>B^|um*N|hUFbH-NGozqWCT!4Pk$)NlGfo zYSZnt1n4k~Gn{5#j1)VB1vjesPs8DL-1Ty?_yR{;TgBQVV?CYku1&+_b36X_*`0PM zvyVi@w?aBn!Q0i7P0Xc$UW|0PrVeUAM_v8_zSJ)6R;Eq%(v5b;cX3}u7aNsO2zA&P z{qUR1A4>f1pl`j`?s4~dl@P4?BFv1(Vc204qAp?n0FTWf8yj<4_(ULlLguA&rg$0% zXmzySH8bwB`%zYiI?s?J)=KnABfXpd?mRix`gXHobMUO=Lxa>*H6an1CQv4UtYlaA z__UlQDKkED*X{(~?qsdr^ox3wZHbv-Sae60>Up{<2(GQ91Sl9Y9gx`O3X#VwL&yWH zAY=TP2+d7ki(P{;P~#O#-*se3JHb8-_RZfXYmQT0|9Xc6A0>D(qM+~OUG}_Z$N=9a zDcil>zFh%OYpmXQql2Qqjn2sFcY2-mIgFcoUZKe^-WMO-41Z;Jjig_p=cvGB%Q`;y zXO7#+Un-xnlFYwLIhhWqH2sV%;J_atxfqc+`wBHfg3Sj;R0#XKt{8h#A%D19&W|3B z66LiY)JMF-AlirbcdMmz>8Q7tb-F38bQ_0ayL^aW0|+6^{z7@N@$NFq-j5?X-^=gH z9DT{4b$?kMQyGHCulJ$ifxTum%XtyFg^4BHcK%&4T@f_a5G{`g`s0gg)4Dw#_fF!? z3NFJ1y~W0$TdllZqQaVndPhF(cHhbcj8a+2HTD3pAc-h?Y$7X}nGx!vem)`XO(o zh-11>qLSdgqjtVGIZ%-%?ZG4*xD7+T>)V zt<`*Pu@HnrEyfIA=rCP*UT~tX#&6ng_AMZiT8#6r9|;IC@6FFbaY9j;ITCZpw)EL{ z$4od}Th+FSw3G`M)u)cyXbr?W^O(c4<26}0cE-3la6&W1F}m=1q;cv!h{k&uh&A1O zHi{AKq;k{G9L7)WW~2!Zbo{qb*zu z7iR++d*cK3Nc^2ehdFKol$Hpm{?bOC_1gg#M7CkQo~3^3p)wwtGCZA=7!PhPo}WKV z-Z@SPcOnB7H3{AG*;Z)uy}rgGBta{1)73nt{UFpmZs$t+)d zqI*d$3u>in(m($y+xgP#Xt@IexZp>E@H-OWIU`)|u=i6a)O-XzP-cYZDK7=f`60L$ zbS~MO+*xBUNEk84$k?Bq!qK%8635ckVxz1pRjETb2s9%QQdvXjh9~V1DC_G@JaW%Z`Iep$M@kt}@CB=K_#9$pff|7qg*rAskR+L|Toh)@MYVWGvyQx2RE7*fEzCJS zhO8!k3UOgs$%s5&fJQ3xEI++hk(E9~`EuaS1*^=Pem)10ff6%}XfGr( zNey7IE9l-8Kg*_Ms&38Pd0ktu)%n`CgLyvwkyZv#C_xt@_RQmz zJf9TpWnL}B{Q+?Tyu#)@b#?>w$>q(vcps{qI^!*;z3xle>k5eHY8WXqUliaP5;|7N zT))L0f$hL4pefkSzSN3J><;FNTopa-G+_lrX+5g-n$6=dqZx*; z02!8sueg9D2d3^<_8oA#(c2xDLdA>q675r(37!?(!b$KAt@~*h_ab-Y3xp%)HKI@9 z{-UCQ9uxnYChNm|IQi@WaEK1`0nv)=_8h^VfI@@Qj?4AO+(b!MmWHLK-TpT5*}vTs zQdZqH1M_;c!=#gH@YQ@88vNRSz`vBFnBDA$378Gg}WzZ#mN1Mu(gKZ&w?~cVi$YlP` zg@`YL`pZ^+r7`|wQpTcpUnSEq109RNdcGLJzmUH|pHk?mzV$0UeeT<)CTGV7T*4mU zWYLx4Nc-h;><__Xt8Rqz_5`BmpWa+e_+#V2KxeDoN`e?t!&TC;2;wiTK(g~Yat3LY zSxik)HxJ+<)mF|SgAikBQY6K(VhOUBGNd92gXyR5;C|Ph20{b{yCgk84x%n~lb5%@ z8Dxa@qr`-dv|8@YzjM#BD3Wt1@-c(Z#M-Ao!+Zr25F@@xu+sAVYCr^HbxD`>TY50+Lg77x zbpx>@8*JHN$xHac84VTlWA_5QS~F_Fi>E)fccnG$qW{lI(m21XbMq zFjmPr(x&7LGGn^dBFBCQA(?J_BR^;RO9Vjc*x*rzkp0BphZ_TL(LX{~6WkPd9{wi7n_Py8HEWL9y^Q%7Ra-hz?A>?it0pC0CFag-IDzm**|72Pk z(qN;JmtssXa|1r*p|SSRD@Ca&wiBcT(w5Ow2dp@}+~>dAk{e@I&k?XRrMuN_!SkX2 zkN{^p5_qdhv8?eUQ7VQv!G(o|CM7Pigj(mM&+Jzo&b=7Faj@aVuW0EJxmVQ(XXQwR zvxKXg#@&Hg(9qn)D&^qmxE9)FqX%B;xa4)hRlLTJ0bj!%N`GvKy2{GZqyswJnys^F zPEdL3RA+60=f1LL6K9nS!QhND{CrSv;8*cbr1!Xj+{0{Jf5^ls4=X09-Ihnp5ng81 zXUhe?r)Q>vL0xpB%ip8RInaR`2>5eOocV!uky<{l7PZn8QE-=dR@%Bv%V=PGjswynWJ!dSy>-d@?{M%><36d zVH?NzadFQMP)n?o^dmAGBWK0`NeA&V>wj_pCk<96yA49B4ogyE!fX}6mZP3U9rK#H z5uUGVrIeJgLJ0G(l(7kQqdvWnF+-{~&*LQ8it=qUQXURD+tGZ?zl1bri_@O{>=V9% zrN9^o-4HG~@diTJWe}^4sP}^C_|fOS&5>*te6Q!hqcCyW@^zbgXal#H7zM}RWD8QvPBU+tQ)%xj zVv`3qoIwO~%Gp4V&Sd4*D;O~jSVU>v+$BK|J496x1Q*aACl?9_-lp zNSY3IJi4qZC|cCVk{(+W77n=e6;(voGtZvx{Qn~`i6wVdOP)w7nrSlm=3U3lkBO%5 zLrTg~k+w0&01OnZrYLznyu8Br-}Q>F05t<3IAE#yW7_pHL$`2YLxY2tf<_90=Z&w4 z1EsUW$Mivn$Ujt%In5Ec|!;UFTr7f4fkwx)UVHYf~wwAtdfcgf5unDujr`(IDNG z6j~%iL2NY=(j;M|U#WcgNu!ig#lC80n)iTWDv|B`!>IPi%tqf<(82~~OPr_@yJoa{ zrfAux9QEPir1l66-cGp$Gje^J7aKztRZVqwZmz*+I<5eOt+%t*lS*zkHM8uza^Ys= z_z{bLD@O>b@g?e_RyP$kXlP$9onv&&)xs5`HXIdD0+3t&c@a+dU~f(bOzS1Zs?6tZ znflgz6!(+&9i8Xlm#zky*PU{mvIU-@<4rL}YT}Z4wwj?3^$~@J-6(QI+{V9ZWr(2r z5Y~MngOQF-$i)r*?9L-qx?NE*Hw3QopV>(L>6=b6-k500uJeTvi{U$CC(eF%+xgB+ z{`f_^xF>ihY42dVtp~v8!yV{h_v!iahk~q6wG*F9GHb!?)a<&5{l~(AMv5jpU5~is z?RGNk{osxYZ1Zibi$nU7yPo6*{mhOG>!4^QC=>=ypFil0Y9SMZV2C{r=so**jIqAM z{|5XADMZN$wTv-8j@)Gn#l8k{;9b*uGxSQe&D8>R*?^ss*jc7%Ko3wn$)|EO;gly{ zgr*yw84w>>rXcd*;4thbe) zBz)hdglSRGYA3+x^NjIz;|1Y{hV|s9f2YFTfOB+lUDY>rpK7{McS+o35HV@8U?i0A zT8ka`9^I@hs`)DSg&uMf7`o!lPA(Z&)|%WCe$`UXw;0arZvGQ-TiNofdA6o zc=jJ%0*h0FRzCU-jy35QW!KY<&uUv-h?xI)!a4JL!dG{k_gUE3MFZ-$Ep*Mcfda>8 znfwow4dhOjP`jZ!rTm3wR+FE7lm`N{uUnkJ)1e4B2qI1Lp=HKIi|Lc~xp8whu5o_z=&un6A!Hi<$VzAp{)JFf@R zenkT{zS?Bc}%A9at zBnyg&UhNIY0MIZ`uTc zT{x;#}yiFEL{9r|78ALUikjfz`@e#5v`hN5Wq3@ama=T?I zueEVt%TCZL8Ac-~Z>PspKLX9;Yws!MmdXXA8ljG#_2OY*6>JxBY@uTaif^mR>Ije+ ziTm|R%ve>>CnTra#YO)!o=7OM0KPn`4jmRoQ%uJ7Tc$L#xu5_Y4!mpO>s1q2^uO`K zf+9R;3;FuAdx@J;wVe}%99IQ!gVi|yZMwe6?7Lc8^y}&PHDH%ZH-SW-gGb};*d7l2 zdGc`}B6PJry&lc$a{A3s5qy!o;7F|Qmx}PwUR0P-$NlXzEWlIa%gG zf?mX)Q&Y2Z4tMKK?NIkUa1o@ydGsZ6IgRx@DREu2Ss6QV$>rV>thLp;ZD7aO&eVT~ zTF5eZZkN)K9|F)N(e&UktHv{U6-Ix%V+eOY?2824-}p4vf=qCIyz!)B{J21^CLL88 z=r9ae@lw0v(Dbf^@d3MTtVWrER$Nl8+by7>xRxT@&_djJr3AfXb^yjii_=YKx^7dc~oj-TdfDab!%az^Iy?4^SE+&V*rGZAeOoYbYQ8Sq- zy6O$lZVD2y`P%g14_KOfbqKa}c==1FQwp0zH^+uSBhOa5GQv*TkQRj(CAD(wm+;|9 zOMKv*iQD4BQdwDgQlxNq_cRv9cu%~G&Bu*~3{IZQiv-POzomCv&>+jp|>$G^Hv1wZmaVAZaS=Yw-J9-7hiIi)MtT#2w47gUX8W(SdW7d zHu%onx>vFSRmviF8zFL+%;~vsuM22hw0SKm9EqihtEVHSQ|O%UA`lU*q%^x_2_~bN zjwjo}@+S()Y4Wx7&)varog>d~`Q3jsPO&z2c6O$X-+%Dp zoBj`pQ(yGb2ZVlfv!7~!@Qd69@80R0>LY1!s>H*LRl(|XtfwJZmvobh>-K+b^*0OWbR)q|pD){f?2G@w>!JBbn;alDF*_Rc zkZPzLj|_PAFBS+!oJQt0J4gz#h?D1iHxflk3cj=RzWL+URQmYvl82k!(2hid&LRvU zh}sQ7WaFY)F7ZJ`G>h?ceknv8l=0X^|J|zo1}!N29u3-MP$g&K!>sSp97i}Od=)8| z*{SWrm>3y>PpYcOP@Ni1pd@jk0bx0N>C`*98Q<3n2#K`LOFZX_h^KQ`7W>~7-iVyg zK`B3+I$A`?H8kb>Z3V&yQg?T;bf0W!Cq{Qz4SDL%Q@LdSe`u|>j8W!4W0R^hI4;;@hs2dj}0m0N;`6VU%kl$L1YYh_;wd{AmrmVQrb# znfkD_d^vml!TnSzP%khivRN8Y03A=fShE`KAU3JJ2&L!X9dwV zvoyl_kiS5ZTH(3Jzs9Wm$xNnCXB-lSf&=KZ3qImj(;mX5GNu)f7t&ZoO&f@Os zq)G08+Iu-970@aMr~7{Yo5HkqPqFC{?J!cbOT-=V|t&QDfLVI0q7T0)!XY&{JR=rcg0+z=3N{ai)N6j^gbb`Z1$BVPOakZ$)L6N@d4`Z;KHkSixsG1asf_ zP(n^~2R|r^fF}C=pO}>@^$uh7xn@}`cA-3jg3PzD(&Vk2p!V^5JvX{fduJmN4v!ww zmFHVza%Ih~rp}w&`SAcS(XKAKUf&2-heZkJGj^qknlii`^uQt%#8rNiac2udv*V4p zv9|Nn`;Q3aYRAJx)j^Q6ze!#25>1DDI6Sl9UTEkxf>{C;D(hZb?tx8k*Ku<(MyW~?z1~yb>NCwdHF)Hf5UQN_Sna(Xt-`#Al-|{SdrVIjWw_orl&RkckZaS= zVk^5Bl1sGM$leg!gXdnuP<&zmFMchAwVLFZ#>tJHA3;WYR@=htAuVjj@fQsjBic*( z>pN$1_xm2%>=zeIlfKe%&&?XQ=wjIFT72;Ul zcaO==*|yUm>rqVF2I#S0YF_}DulFW;M!EG-aZH+kHib8qtNxFLy`0#lQlI(kUvb&5 z?3?FTKeH=Y&Nn5c;O(916Hc!@dtUQgm#FreATRD_cK&VNu}Q0@deyt_)KA+U7PIQ2 zeSg%&=U1*@`mQjkE>G#z_Z1gE{8s+nqk6CT-`q?8m)B+)?r?VHJ4WDUXu0Au zhIzXAx3}fy>CTgx{N#p3=DJ|cnvJ%9zgz~cmHaQif8KY+^{b!Nx*QY_c&DGYyD!VR z#_{E=!z=VxWLuzoMxnH&dtY+`BZiSl&vW@%aAss;@tycbBb= z?pDiP{WRy+mXm9zhIOCUdO5xBQni+Ay!NY_uRrbxFf0)f`Tp7cw{p;}xG9D9!ZE@3 zw>xO=yCQGd`)jXiQyZH_?5^EEpUu7-wJ$L0gdsCK-xlu+$4|vis%vLA+0+e8pUbn} zIsd<^ch&CK!|b1vRz73-P}` z6?<3vFMV;|3FHb-`|VqRJ9zS^?^qV!<&@yeE};_Z`ojB{@xk{z??TIi4!v*Kb5FnZ z<@wEV@8@d>{#_c{@2A#qTH)ofoQb~|>Ad+a{4I0W=Z5($^>&u-vtGNL{F}b*_$xow z!X+DRUfo#oU9s|4y4d%Bg|=&F&7Zu?n)yfXzok}IF0t{SJ`{@-et+=%^^^O{YxDv= z|NohMPxwUL&zrw*E6m^MSET>%=c0AJOYFA??Kdu5^v+TJ@8(H+iuU_Ity2s1JW@N) z^PYVOFg}C==lXDId-QS?EZDE3dK8#g{|o>8FX;IG`u~%cfhA9ToaevKQ-Al*dSa(? z|NqRN@9p17UHvns#m-&i=S-KqVZceqJ^X8ae^)$yqEP8&y6W|VKfj8v{Qs0Q^Sx)? zw}0C%JU_hk@j3Qi>6*v0r`GLs|2t&jM;YZ?0(S1y=zIix?o@jNS4u8%H!)6;H29WK48y}r~`MF_$3GK3RO~CwCqv8 zYoy47Iq!j|{9Kh&cTiq6fA`%>U0k3#*0&$L=?k*iW`f~=`DL$Wn}50>@fCPt5re0z KpUXO@geCxLdSVy= literal 0 HcmV?d00001 diff --git a/documentation/test-manual/figures/test-manual-title.png b/documentation/test-manual/figures/test-manual-title.png new file mode 100644 index 0000000000000000000000000000000000000000..c709cb9d09cfa38b0909cdf70b459946e3b4615b GIT binary patch literal 15382 zcmch;Ra9J0^e#vYPtZW{;O-VYcyM=V2sG}l4GA7BxVyW%h9JS+rEzcEq3PSn@4x0@ z*35mGb?2cePWL%gyLN5)>f3d~loh4lq7tGaARxSz`68i;fba?f{{JuJm+-%c^+XHs z=9ROUj5;zh^74k#I{cErMN->E&ECw#-Pj3;U~Xq`3uJaSaRLJEoGt8KPG5BiBOrW4 zkdY8o_eh7Vy6fW(fRWBx`X^%y3x3c^zQAJg|9VR3&Be|h#?HQT-*=VyAulY9o`$_} z0h`;+#AX2#t^fPWA6#f3J8qqz_pgN9s+f@UDM4$ju;z@k<%|}Cl+)3Xv076Of){XU z5V#kDS^l>ED&qC8;cfTpN$kJZI1+My8+lCTzqhf8{(t;Y_hZZ#bc5Fh|Mjz~s-f|o zU66*sEk6e>mr-9F|L<5{)u9{g!7Sg$a_mvym;NZZ)PKLG2~GUlB!AR}E0ql)=8L+a zVWshI#d70OJzF<=T{~RBE8G5+`yfflz@OXsmn3GLZqNq|5&OH1y|;RPkJNN*%D~M2 z{{C@IKTm#kl}v0Rp{I_fHJ9q9Q(ctHiwkG?=i*Y}x@1%jmTedOju;bRNi6pf0PV6#aqpH!%a2n3O`gpvj4k0lt*;KdoIIUMh*?*gbbz;)w5}Oq=}yjHldx zxUPUdnTmlai5`s=-p2s;dW2#Wji(fCfLga^4yKh@eN}#sz_r<57mFjc37_T_WE>*U z34qo4sI49sq#YcD({H@tPIICe>E%gB7ZhsiY7+~1bdQY4XI}h%xYlceNsewEZaP_jN#Dxl z3he8_S9+p}7M~^C4%Bucov>J(Get2Zo#|^#z^w;>2@lHbmE>TPQwcIL@g1}`CDC%L z7<;a1~)PkgU`(o=XUFt5XEF*V?3rwQH#A>S}5hUiY}* z{j;xEcg|b(7xT;R5Y@44OzQU5av>+#Hcz$ZrO{c}fyB|ns9N^{H+R{6HEneX-y7TG zh4Yyat+A}x;{A-K)9x3CyY?c`^-AH7YRJhnFTpN67@;&EWnS_&(_LnsJi@f znja2MaoP;>_{V=v_@V!~YiT~cJp~m0py{Ug&ei$aoN>F#9%wcM2ziZU>6HC(h~kY5 z%+Sg^#$gwoJG!Z25@EzV%o9&pV5?dUdb}`UCH<)KVH53#S&^%6}3TIWxG?K?#YSSKk%aDDU z<8YD{X$xyL{KWmAfFlKU5XaB91_sOFJ`bo%Rj{nh3R|bBs3-$r7g^oZr-ryZ@ub`7 zl2?{2ZVMBNJAO`DkHwqL-l?q)5{Ul^6}B{w&1;NJZaVj!cAGu`DTpXrUe|TY%nM%_ zXlXgQLpj%E<>^ZGFE|CeoNPPZ9|Z>^1`)`KR6S;H)GehxudJ-(CP0h1j8_(-IO$uQ zS*z6U^8}X%xSGZ`*td^g{`Y9_hC^}Fw3<27D99$bRH?qr>AI9AL^-s3Nj`&1=DAfp z$(oQ)fX{V9&F^@D+z)cJwQBr4FP&_cEET+rcM6)C8d1+HQAzz5G>wcR4F8B7IN*ov}3@uJ|14!g zfl$PdoLX!&q5vo4ohq+A(3I_zkD5E7mB8=pmsrdTU(0WOXDTLK^yc#(%}wr^5P>_T zko)(UM;sJ$-53%73E{yg{P39^7?4UJx4XMLGg_3$1WfLr!V6Doze45i`;`N|;XQO# zdHwR2H<@3qqo;s5OmS%%ocmRp!}Qtr67SHA3;-aBFLwTKG2d8QGu|KRa29fI%ZG}{ zu15SIA@JJ_bZRsih{hnml2+i80Ni<>MnPsqaf%eB(W5PG)H@6(7ccQCf(N*`xY%&z zsjvqD`1tt4fV#!wUjc}Nywun-y=Td(OS(Hk@@YcuzfXxu=mlUaLk<#9qd`;M&S7}$ zL*W1G$dp7u<^}Z24||PdJWshl_jeWrxTpd#V-rD)(VZ4J}!>9D&J?z;Xs=#V(CO;8y z%5;&r88AbkRl5vVgql>Oa6Ex?_FsN7@|gKT;qv*nBY)% z4>8aJp3%C`iYHY=NaUPmXJ@wxIU*Mu`0Cx8%E}nU5mV&xc#UIQ*+0D@0^JEyr1QKo zXqo;pHq}G%H+BJGU?eTyYu6Cz$tSsqS0jJJBO9AlJ-g@k%~416s>de-1 zf8wlh*uioGJVO3W7Rcd!vVtIZkbIg6!%;U|zr0tYRs5pJpOx-nNDdE)RJ0jce72=x zo9kz1@XUz{52~iS5eM+6_eynShkma@!M(5ZP!o_J{^nrxqu)we!~%}0ot8G=USeMT zg$5rccFMF$a;B0PwLVm{Wk;#VXTqiPcy0S=z2`3SOagR=peb$bcF3M4EXF9FHVhwgcK+% z_3rYY5N4-0A{`waZw*lQ5&p|gMa|6~OX;2}0hFDOR&QNntl7-|2QwJ`&8FE%d6lm~ z=OdOlI5>xGk&%&<14{AsrRg6vbanZGbHw+%=42o?m07P%tluUNw`lN$a5~fb^z>MV z#}scL6%l#wbvpRbkDY0woj;|&Q4+b_FlW>Y`aV#o zFx`#-qoJwqFsji;wQqPM9ySoIqpQoq%^jJ^6-3A|DAKku4`@1BSiDL7yR zG0!SesRnZAfXcKAz!o?h`$rVtf9~`P%B%qM+LM>^r+X3q3#~%SA*by-`|E5r`^v5a zH_!&(xh=G?)b5>&S<^!^Pftm6w6jtCGJ(gK%)Wj8HuxN})ewV%*aKw)Fp*n` z8Rza|pAsWgYU%}2P{<%mM_ZfDRJ&ty8tyS$N9z2fmeU?sK*;7%DJUhR;-)QuYQsbi5(L$Ve20#d5ebJuZNNu&+Gd=IWuCQzk_fC6h1cNF!pY zFt-j4P!}ncT5>R#^xR!0BS^8gtnuT zYUf@olp?2avol(StZcLdRuO{iR8=28=>1LzWG;BNZ&qEhFdl4$-q%-Rz{RgAHD?Mt z{DDDLc--oC>IBkI36IAV^VnTGVG$2_wL1``cxYRw6Gjbm=j5STd7|xDHF?l=WJ-B%N%5OekenZZ%i(^iT?_xsrTU$sU8JpRz zNCCA(l(*`E77Ulk;L!HkNKF$Y8%-Ly&;d{T|1L|7-4!&r@K#1yn9I!-qccqv;#vbYs{49BdGJ9sPODTq zG$~SUw-@JF!|qVcayPNcY^I%aULMAVqFv!;&F4iW^HKDtKW`bT)*ZAlpPE@TQW*K} zUNS}zTPshl52jv%p-xuR z@|bQR+ixikSMam_0q+`C4+i=H&Ovtt^O#7wfn0`wMvBWBBxX$yF|`JHKuGYxmd7wl4W3n(hbAyJFZCseud&DUZnx_zH(;B||LEfJJ8MHSk z@@>;v+`vdAUNT_`J%jIe@1lI|d(e18rrJ#ZMuvrT>1*@X+?gVZj? zE151DfmmW+^1H@okZ*PDiGPNu<&M;etYJWhil9@Fhmz9~Pb36?>BKn@&!rH2Y318m zgsxBT+#9K;ciXhz>64gCpK8bA0fd^;I%gt}_TTu8gBGZ8#I*IhHq@e6F0zvve0;^0 zjQ~V4c2j*4OUhA$@q1)?yVO||!)}CZW_vc?OTs6Ht6FP|Z_?q*RS}0r1?Z4asy%mc zEtHVE;w6ebLf-JK)W6pp8@Oy@%6J|i3TN#k+`symv{p*r>d4QRmTH7`v1P-mCu{5( z^=h8^EPK0nIZ;_G1*PaeFQJ3H<)~(&86{duN}c)%J+fVDeUTF#8IkACgk`xE7Wi1N z84kHWU?MBXA+`gn5DaQGGm#S}nd$mBF}fb?5MfCi-kT;0W1FpVWy{f-{WV0wM3 zKnMY!2Qskj%4^ORn&kxt@9LJ$>>oj=#p6f@>COtH`9|R{%A^#!WB$4zBC-m zv+4{7>LbPQ@E9qts$~Fro+^lTE6k^p*X20(7XFiNi5Rn*9gV10Ldpqu1nTRcB8xJ) zpPl6IhOkJNr(6RmTd2l$98aw}T(UEjgy3<#w7%7aV)*OOQ;u(CMn~S^=nGD%DAi-m zj}-~G(+5yY;a?|FrFv{}+27u6JF!z2@R#Vf_pSwEQWP`&DU2?=yJ^yztmtX*z6d)| za#P1ZTw-uY3dN{;(7(K&&NPu$tT!7v`GAL8{($o5q0G!$Qj~@Ho3%Yng|(H+2<^^Q zBUQdoWoe5xIY_aqh!g>VR7cm)Vrhoc3b@dBPcKW!jz?Z5U1lNjMx&tb3Kv63j(e18 zxqopk7MHiNsD4`=71$dn8JgXi{=AKv7#`k|x|7=`srP2C#n8BG#KoLAnDWK1n?q8o z@+K?PLo3TBdF=C=25Siav%a%i(3d%-s{aOEA`LatQA`u*)uQL~&|Uc;w=<`88jw*;BIfKwRmoXP1ENPIpAPMos~x{&ggWv9kg$ z9ssxg$1=?!sxI!$Z}$1Yt;KpxBN8X{=lk#?@a>lSOvCTGJvicrcP>x2zZq#6d5s}l zQuQxiu#j*uF=b}-*nEfzt0>G<;@HhZEO6Js5H7e2mM<*|(Q~Qj<~QD4hi>ODU<^EZ zmK<^BZfF-{_{Z;;rv4Zvw2-H>5%`TLc#VK?R?K!h?vPlriH;)^@Wob-_B{&j$pZ)X zUM&iFprnq*`7aP*U#kUYgT^`wYAV-`D{CJgpAF0S(o$=5h+KAujCpXjwoOCKkG0Z4 z4t9;^m!tT8tv+ggqqIFokbg;cxOx(Md1!7W5OWBZz}T`wTS`5eA}sCD`hec;#x|09 zxipC&4RdHey1HDHVj83r)rh*nt)DgVEnIwdrN0pFdo;^K{So5GKrTh~B4H-A4kV$B zzaHm8;L$>$=8ezmC2I@{`0Q$E2~=_AWbWhmbN(wC2R!#HEVHe?Sq7g`)T*poO3{9n z>i~HAMGo`5CYk_ugPi?;mVlUvIqZIh%FdT1ebDWAQ)3xSpNmWHqlG zr_RM_?h@i_{Le_ihC>T0u7wyK9j!@T^|F$C-|L0l)$pLt^yAMe7+TaudaaS}1s!nu z?b0c3>h*f!lu1y1{Mw6m44OP1Ultftn0_Rp+8keLC^QxZHPYc54v#_V=_cjnjEvlWTUP&rd#YI~EhxV7v#Pm)XAY*VyKRp z*5mecn?{_}vSv_v<~KM;u^VlX8Hf5fOfK0vp2D`fTpI4bM-uJi=-n!H(Cm@G9o#rL zDiL|v)?q~OA$7Ws)}7^(u0E}OrRn#0oJSh6;#}Y+QoF!PqSF>0FB^*!0&~dK6q&cC z=};Sa{;ZDqp6yssMe06w=S*5H2Htooh%tFxYHExl=*_o!4I|2;CM%}^>9@H;5h+~u zj~nZSn>T}NQ-$sCMPPD}4;*j9(pR1*TJ5sDBL9KfR&NKn`wJx&z0B%{c)mAONgwiZZD z<^+5&d!>l^d^M2wvSgHMeq}uMr<3dtDp4+mEw996t3D z$OG;pBiHkOy0K#33sTlPCu_C(31n<<)0<6V^ENZCnCfc6&6wJ2OuR|BHGF7=po*e_$0sIcKI_tepCxHUhSUGyq=4vNEHUJb1>wi8zOq-`x^WXTA}8BvDlbYA zy+851AX+0MGcVm_?#p-`A$;x>8nSssXlG>O;irI_Q}B{Vz*i$~??dM$#8{N@N=7G0 zKEiKkUXkAXRI-!Cj2l%+B~{ydi{D`Ij0Hh)wIQA)|W z8JjSgwKjD-nz07KH?-?mrs$N87>>3ySOepep{&v)^|_{xnWGf2EE!WwUVmlv>+hHN(QnC4OP@K=}Q2C~y%_^|g5uj3dF)MowxHFUZAl6(Z+nRwUkqdeT9NawjK-*Z3k z-A(T|8Mt3d5Db4IeljN~g=ZR4c;#e2QJqTNt6M*sEd4b+_n_udPtaQ`SSsK!2bmbl z$L+u=5`H+H>heY3{q#tj`iiSFoS>2)P94acagw=+dbs@_uVdv^=RtW7u5m!E{m0G; zp)4^^yh$mR(I}P(wFGysXEqk|!2!f3Ci6GpMW&R+ffsS`GGewgoc{It9hS~%4+KPJ z=D~?fBSAO|>GI8X$H;t~u>!ln_QLn&gOGs9%!PxV#&{Z>0A;h%9wved<&W}@jQpQE zC7kN{OZomk7cu|;`tfgFHA2^)vh)PJNdLb3}) zI&6Q+9{f&dabeUXO^S)ZyDZw3_x3BUDR2+ueqWP)B*6BT|9ZH6(rk$l%l}~^R6_xIC(wtL5p@z0LWIxeY{O|k=cWvFD8%t!eg?RB|I#8E zvQa1d5ynx8SHFJ@uiEoZHFQ!(*&hdH3F#FKoqi#DUi7{3z@ZL0z24m-Un>P!rMB@C z{I1_k%#8>vjH+|h>~h30Jg7LBALek~k1kQo#^-Nc-do#P)DK5MAjcgVWnkZZ_bVVKO74e(qxeuZQg6MRfd7Q?s3*BfFRDi)>o9R8Kh$ z(Xq(1WfmZb9P3c2ja58m5BKHyp%4qhQ$Y<>EZJU$CJPMuW=M<>49R{r#v|&srMM35C2i zICsR=53rV~`cs3ICh%x`gU_|M+MV|9@tnSE2N)@!>C$x|YvJcHn%2chY6q58Zrt1me7Pb0;7q zW)op?QMKpf)-rGEB6EE_|B!U4Q7;mLnZkBC6{Pg8h_p>@`6DIXO=Vt)R0mAkBLDpk zMJ$RPG_C$ZTlWc)s!PA2y)Yp04&-My^Wib7GO1GkaJ8z|j@853o%N_x0BB1zdLQyb zG7su8)KJ`!}gXpNo^ajB^}pXuzv-kDp!4MSncI)wG1Oum$#{6<$&I=6;C~z#eQN zGp8PZvveB*8qR}_;5Kcqjo$Crjj?*K8u|jCwNyH0u9d>9PJB8+up-MJM)GqQ9=IXXuf1*DTvf^4g)?ZLp105Z#HaY&cGIyr zHFHCg;{TS$($?xHwiw~_v+tHkQQ6L*f(A8=`1vslt*Hql>>WcHf+?%pUA z5znPTof)Rop4@U6N4usnyNxZ@>R4i3HpFiNAux8^`x8JQb4_bE7xqLP-$SkPFiVl+ zGrYR2HgUe{$RWDZSkac0$)J%8p|Z98^XJR>pD!@8rrSqY+T-A7xq>dB)I))5o)m%+ z8)DWDbJ?gc=S3v+2Lvgh+GBO;?dA9!vPK7eW$XRSvL9WDtV;!qbQ!0WfnMRG$%q+~ zHZf^M0ajzgYm~P#czxVv3y4`mp)_IQ*F`B}?ZGe8zbN)prRNrtYj{WHmnX_NBpAtx#z3Vsv(k z48s`z!@P}b@l55-oEiWyf{P_4QG(P6KlLy~dI5gi+#i%mNYZIyakbUZkq+zy<(fa5 z?an^#4Al`z&JJ)aQNFJFY|ii?xmu2e z@Zra|JnH|8RZDC6A6CuEjB2*arwPWYF`8JLID(_rv@?4h=n>YpGX)z$OFcSizj4;( z`)!Uo-r-TI`~2pd_(9%1x5JNDL*tbcGL{Be$E>`9(IWTZjzrSFP6BXrAW#3W+e!pC9>dNR# zFHsmB2b0aNy5@4PR{ou-=%d&FP;Ri89f40b9V*Y{b5lJ6O>-xmPBp$g9tS@ZER{gG z8B4sU+Io-O zby;{;Ef24EIBz+txG@eZa1q0-FQPOiRD1F|GPp=<{t||cXvQXoLEq&Y7O8I)A%MhE zTsiHQ2n;tzUaZ~KmN;bzzov%=HNV&9%y|mlQ$4qNW+GbAnS8%z5>cWu+nuLQj)DD} z$8#m|dTJI(htCCT5Qd|h@A#;pw+KpYzOxwoI}$-iC*fwVRi$x(vrU}S%e)yho(}eG zZkeG&RjkofdDJz>X7b^w8@7G6rJvj`o^E`GO03$??Zj4^4{m1c4fN*28eO{J{zMlX zk@z}j@Dz}9?RA^-G?E~>*2pA=9Cme^LbxG(6#43cM=IX)X5cXgaQ|nRsnwGuZE0dz zYE%Xa2&FBEyx$Cax)olz1Lg?kTqQXtRE;_B;xsOI=VYN%NhoXk@+9y_F50e7D7}Sk z?q{Dt$rg_N*F0C%k?GuH?0atn!u?6bOuh0gvfW^)odSGEkf<9aCn!j zX&4@gK68mX=DZBziNM_C66vkQ!Qs`u%Kb6q4zNg*9l1t3B!YaUu`TX2>9JqVDL-FB z5_sRL;KA;kxgan-a3K2bd^J2|z}F8-N-f>Dcn2_Vzl#!RP+oOkL{)TW_Xpw)XjLfiOSYREQPt&~`(`&B$dDKPlb3Inoyc{jy%hrHaHz$Gcg}?C!2P#FEEf*&q z?q3w!1B-@@@hq=1x4K=?1nKw=8>!>#F$iV!I4+8<-zSf!PmRYt(?;2DT(50@i^qDY znx>^v7JXuSp0RCx6b+c%2HPhMzxQ6zLG=}gMMc@T&Nw?oGTFP=)^YM2GyKK~Hn<9| z>k%y*$E_b9sl9p^RoL)tbkA;vPGJzU(6Es8=op*pR}Lp6@Ti^opfX~ZHvoJaUw}IGDNN{6>d$?g-ZS@=!sOxw5OvH>hG`~{(!%Wl_PM8% z#hqLgGsp<&y$&LJ5*m+Hb80wVLDJ}T%6CWoHxa~wrY1R0yEjuuI!*Z3qziMr%s3j8 zsmc@@*YoqG_Q&?DH^z&L4=oG#{Mv4X%lUe>ioIR=#+u_{d6enp{B+4sZ#l|WC3#;8 zNr(6Dd|)Ol1v-iE@wiY!QkzC6%yc{iPsWAib)?tr4xtx(V;v>TZPlAuHcNlBPJcn4 zTD?`0S-m8fH@ zUrAYE0fRN#0E3%9H@%1UbWeHs_3sBkbQDRA7TV>aG3Pvr4Cc_%k}F8FC=RWkQrLTs z4j;scT))P;Dr;6%#RHnME)_~0i{C!?glBToDv=1E$tIoI#fvV1@n`#G6U4huViY#LM`+C11B^>>B`Tm!VF8cx1 zmAwJyh17vIa3JbecC{m`_=qV_o+;S<3E&QW5i9?Whsp^SACc0t017D`l`67Iu>Tql zl9ffA0Jh7;5IW@Rnf=MEH9%k5c25DZ%G5fw=mhK#Rnfuev#wh>d{j8UV$T9VTjA_DR+q~m-ZE>ud8aif)-^SVrMaEN5zA-httzl-^?0;# zgVQu}w{v0~QLGAS@Xde_yLvs9ZMijM37mI0gojnbMjnWnxavQ7dT-j9R%WF%)jxAI zSTY|cLSqvRdId-!eN=7k*wl|9{XO$rb=KaIrzl_tlrV?YoaWSJ6IExvJoXxhWT%$K zG*nwS=wo4DXzfjYm0zGYhBrj)5G0VOB3i-2%Fjr)x z!vqse68HPU&Zp5|%t1?sM(lSwPgHxwN@&Gx=$azkoR(Us2rc;w|ASVSF;Ii;KG zoJbg5Er^Gh?EbjCv3RK=a!}r&V>~M2nm`eV$ulou{1ETY(O8f_432&oRd4~){=utx zuD~bDZTjhFB8#i!QfaN5V>{AA3#8O%Axztm3vgCT*~P^vP$U#Sz{DCk$7sRaDhOH4 zLs>fZWr`VM*Rdz8bm4E{DJ1-R0?rL}Cv@DhUkqLMA3WM$q4nc)=@}uvAC#l|2 zG*pDEQz=_$%8#}n`V|`|SzM}~zK}OCyWTi%z0X!Fr^~A@*C5PVUX#Yd+}J7q6=u`! z`z#kEK$n38c{ZA|*}HLEn`_5`uC@NGy3)yr8&r35mhJ1&)Htw>Xq7^kz!K;FY{N*z z={3z%ss?tm@&`h8VLe-QW_=D28?*UNRkD)fKMv%Ao8Nv#Hpi5m_q87d>S^qxb*fCpu>b&%KvS%kpDWm zoa=<}#b;^s8|T&hBq3icDiz}fWg<1Q#SZ-rv&iVj`$+2B#&-9kib>AngVXsP8ujNU zP5gZXc-XBL;m-L;+mTS8xY8-%Ym?&JbuiW%DJ2gvSlOo!x_@i6gb1$(aobW`ewHguznU}1c+gouFH!TPGSCmr6P@fwr zU0O9Yw^hpb%vhNLFc&N5Rb7u-b>(!=CjZ#o4}=R!1LER7NL82*bnhvxo0@F>akF^%XC`#$r<6MR}fzi3bK}JCA;nC1WZ21mY4rq02tk3<@Uy)P0zPHhZ=txYiq&dY|FF#&M)$eJf zL4BH9VYTcn@g&uZTtEc@tfGoMRxT+dAVa z^@uMBwnU`9O02Js)(w(qVHDbzqAt)NkQ57&tN<$9`nBX^`u_>!zbENE0ajsdYQW;w zvN2i080~z$<2KHuG$;}Fxqd?p_ZAh1OglD_ckXp|1_4DF%YahZ?zV6l6i z<)hmyJQ3@08i{mI2j`iam-kcghOy6enGPB?QQ`Fs$v0G5okZ6>p71hO!oM=s?$o3+ zfft_P1SVNp9Pdt@GzTkqDMjgCZS>Y~z3fqDfGNBtujA9AC&Uj1e3J#u;63F799{y$ zt<}x`YeMcplP89Ux&_f_YgAlA8Q%0-C69d|UB6MhAHd6FTpAKhycMg& zxA9eVEI5WjE-uyqJp&RA`~(}O0?o^**h|hCYVT+aKCpY{Ke2}8+YRi-C4DfqR0cYE9}?SKDVAWVP@GAs^g)Ux|OQ`yuAU z+I6m&N)lu;IUXa9N)Kh2T8VxhOKbi!kSH0Nf6$ZDp?0f9=PU08F9#X@ESWp?XmGfM z$ko~_$CXye-}ksIPwms&(I^zS`#oQtx>unfv)55D|9GK9L|F#+#6RUhKH zV1&T@PIl)(>m#>b#At^dv{JdK%yId=F$o$T)-f~(&xLZm+elptZ|9Duoa(9Aabf*7 zUM`E+u7I`z6eXI737u zypK07p1Z3c(%k^OgN=7Jo7d>&J`>EAdp#<4%5)3`HY zP7~GQd4}t1mREi&^4K znJzx2y6SAh?RU^IfTQbXf|VFD>v7kGvfN)H6hBs_{7@+AUKH;kwX6S}gwq*%ot2nB zK~vY5xO2ihta1Nhx;spO=KATn<}Vc$17h*Wg!E#Dj}H}#uOI&s*RSA)3ND){Z~qxZ z!aj&`_BB2+mW8G51zM6x8!S8L=5}y=f8sv6oX1?g?E(R_gKnOi9U=G%*AG`oK84cD z=TDmB`@UgwSu}aL;6GNc5i(5I-@rVgHUz+=c=^ASXv%(uoh#i>6@}KWhKMxU=LWP! z)75w$`dz~ID$npkh&~Xl0bHM9*o}G)KM$?EUPn*&g+}ueFPp=fJY2J-a{NpDRt909od@R#t-|cAzCrGZ_q$Jm7C&$1c2miqOFlF7bct34LW^H#u4prgrq7fx; za|<@WJ%QowR;HnorDL_9^!0ihhW71!ZKQa4f?vk@zhdZ;^B$&k=-$4F!YtSd{HN3K zp+azY1e$?i$R{{G9z>rcxpMU;C978w<}O9n>Rw_U4OcrOI2+kYGUJuh&EX7y$Z(;Y zwh4X;@;*}o7K1_-L2T3iRp-I?A-PPL1q)}d39NG#8!{xG5@p!zp{37CnpReQQ5br2 zO!&_ogV|jmwKhJKO@25RnitoR__c~=Nv5-?@RKCK&BGHs9K5HM=4$q_LoFK96iJ2j zG(HNJXP?8&BhREuMyE18vK8gNjkcCgLATc#Jc2Hu(2j&wgX%jK=PVgVfyZAtGdP}+ z6@i$IVc~U`>z`qKdGLDx9cyRDMIpK%Pt^@P;897yd;7Ubq{`9l`|yi?{1b&QtG>{+ zf=7*=mtud*3isZv(dcL(EA@3uJq5B)^Uf}(s`I+^SB}5xn7zl36~=^peuSuncG;Uh zBDZwlwO=_&% + + + + + + + + + + + + + + + + + + + + diff --git a/documentation/test-manual/test-manual-intro.xml b/documentation/test-manual/test-manual-intro.xml new file mode 100644 index 0000000000..5e9def894e --- /dev/null +++ b/documentation/test-manual/test-manual-intro.xml @@ -0,0 +1,634 @@ + %poky; ] > + + + +The Yocto Project Test Environment Manual +
+ Welcome + + Welcome to the Yocto Project Test Environment Manual! This manual is a work in + progress. The manual contains information about the testing environment used by the + Yocto Project to make sure each major and minor release works as intended. All the + project’s testing infrastructure and processes are publicly visible and available so + that the community can see what testing is being performed, how it’s being done and the + current status of the tests and the project at any given time. It is intended that Other + organizations can leverage off the process and testing environment used by the Yocto + Project to create their own automated, production test environment, building upon the + foundations from the project core. + + Currently, the Yocto Project Test Environment Manual has no projected release date. + This manual is a work-in-progress and is being initially loaded with information from + the README files and notes from key engineers: + + + yocto-autobuilder2: This README.md is the main README which + detials how to set up the Yocto Project Autobuilder. The + yocto-autobuilder2 repository represents the Yocto + Project's console UI plugin to Buildbot and the configuration necessary to + configure Buildbot to perform the testing the project requires. + + + + yocto-autobuilder-helper: This + README and repository contains Yocto + Project Autobuilder Helper scripts and configuration. The + yocto-autobuilder-helper repository contains the + "glue" logic that defines which tests to run and how to run them. As a + result, it can be used by any Continuous Improvement (CI) system to run + builds, support getting the correct code revisions, configure builds and + layers, run builds, and collect results. The code is independent of any CI + system, which means the code can work Buildbot, Jenkins, or others. This + repository has a branch per release of the project defining the tests to run + on a per release basis. + + + +
+ +
+ Yocto Project Autobuilder Overview + + The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and + procedures used by the Yocto Project to test released software across supported hardware + in an automated and regular fashion. Basically, during the development of a Yocto + Project release, the Autobuilder tests if things work. The Autobuilder builds all test + targets and runs all the tests. + + The Yocto Project uses now uses standard upstream Buildbot (version 9) to drive + its integration and testing. Buildbot Nine has a plug-in interface that the Yocto + Project customizes using code from the yocto-autobuilder2 + repository, adding its own console UI plugin. The resulting UI plug-in allows you to + visualize builds in a way suited to the project's needs. + + A helper layer provides configuration and job management through + scripts found in the yocto-autobuilder-helper repository. The + helper layer contains the bulk of the build configuration + information and is release-specific, which makes it highly customizable on a per-project + basis. The layer is CI system-agnostic and contains a number of Helper scripts that can + generate build configurations from simple JSON files. + The project uses Buildbot for historical reasons but also because many of the + project developers have knowledge of python. It is possible to use the outer + layers from another Continuous Integration (CI) system such as Jenkins + instead of Buildbot. + + + + The following figure shows the Yocto Project Autobuilder stack with a topology that + includes a controller and a cluster of workers: + +
+ +
+ Yocto Project Tests - Types of Testing Overview + + The Autobuilder tests different elements of the project by using thefollowing types of + tests: + + + Build Testing: Tests whether specific configurations + build by varying MACHINE, DISTRO, other configuration options, and + the specific target images being built (or world). Used to trigger builds of + all the different test configurations on the Autobuilder. Builds usually + cover many different targets for different architectures, machines, and + distributions, as well as different configurations, such as different init + systems. The Autobuilder tests literally hundreds of configurations and + targets. + + + Sanity Checks During the Build Process: + Tests initiated through the insane class. These checks + ensure the output of the builds are correct. For example, does + the ELF architecture in the generated binaries match the target + system? ARM binaries would not work in a MIPS system! + + + + + + Build Performance Testing: Tests whether or not + commonly used steps during builds work efficiently and avoid regressions. + Tests to time commonly used usage scenarios are run through + oe-build-perf-test. These tests are run on isolated + machines so that the time measurements of the tests are accurate and no + other processes interfere with the timing results. The project currently + tests performance on two different distributions, Fedora and Ubuntu, to + ensure we have no single point of failure and can ensure the different + distros work effectively. + + + + eSDK Testing: Image tests initiated through the + following command: + + $ bitbake image -c testsdkext + + The tests utilize the testsdkext class and the + do_testsdkext task. + + + + Feature Testing: Various scenario-based tests are run + through the OpenEmbedded Self-Test (oe-selftest). We test oe-selftest on + each of the main distrubutions we support. + + + + Image Testing: Image tests initiated through the + following command: + + $ bitbake image -c testimage + + The tests utilize the testimage* classes and the do_testimage task. + + + + Layer Testing: The Autobuilder has the possibility to + test whether specific layers work with the test of the system. The layers + tested may be selected by members of the project. Some key community layers + are also tested periodically. + + + + Package Testing: A Package Test (ptest) runs tests + against packages built by the OpenEmbedded build system on the target + machine. See the "Testing Packages + With ptest" section in the Yocto Project Development Tasks + Manual and the "Ptest" Wiki + page for more information on Ptest. + + + + SDK Testing: Image tests initiated through the + following command: + + $ bitbake image -c testsdk + + The tests utilize the testsdk class and the + do_testsdk task. + + + + Unit Testing: Unit tests on various components of the + system run through oe-selftest and bitbake-selftest. + + + + Automatic Upgrade Helper: This target tests whether new + versions of software are available and whether we can automatically upgrade + to those new versions. If so, this target emails the maintainers with a + patch to let them know this is possible. + + + +
+ +
+ How Tests Map to Areas of Code + + + Tests map into the codebase as follows: + + + bitbake-selftest: + + These tests are self-contained and test BitBake as well as its + APIs, which include the fetchers. The tests are located in + bitbake/lib/*/tests. + + + From within the BitBake repository, run the following: + + $ bitbake-selftest + + + + + To skip tests that access the Internet, use the + BB_SKIP_NETTEST variable when running + "bitbake-selftest" as follows: + + $ BB_SKIP_NETTEST=yes bitbake-selftest + The + default output is quiet and just prints a summary of what was + run. To see more information, there is a verbose + option: + $ bitbake-selftest -v + + Use this option when you wish to skip tests that access the + network, which are mostly necessary to test the fetcher modules. + To specify individual test modules to run, append the test + module name to the "bitbake-selftest" command. For example, to + specify the tests for the bb.data.module, run: + + $ bitbake-selftest bb.test.data.module + You + can also specify individual tests by defining the full name and + module plus the class path of the test, for example: + + $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override + + + + The tests are based on Python + unittest. + + + + + oe-selftest: + + These tests use OE to test the workflows, which include + testing specific features, behaviors of tasks, and API unit + tests. + + + The tests can take advantage of parallelism through the "-j" + option, which can specify a number of threads to spread the + tests across. Note that all tests from a given class of tests + will run in the same thread. To parallelize large numbers of + tests you can split the class into multiple units. + + + The tests are based on Python unittest. + + + The code for the tests resides in + meta/lib/oeqa/selftest/cases/. + + + To run all the tests, enter the following command: + + $ oe-selftest -a + + + + + To run a specific test, use the following command form where + testname is the name of the + specific test: + + $ oe-selftest -r testname + + For example, the following command would run the tinfoil getVar + API + test: + $ oe-selftest -r tinfoil.TinfoilTests.test_getvar + It + is also possible to run a set of tests. For example the + following command will run all of the tinfoil + tests: + $ oe-selftest -r tinfoil + + + + + + testimage: + + + These tests build an image, boot it, and run tests + against the image's content. + + The code for these tests resides in meta/lib/oeqa/runtime/cases/. + + You need to set the + IMAGE_CLASSES + variable as follows: + + IMAGE_CLASSES += "testimage" + + + + Run the tests using the following command form: + + $ bitbake image -c testimage + + + + + + testsdk: + + These tests build an SDK, install it, and then run tests against that SDK. + The code for these tests resides in meta/lib/oeqa/sdk/cases/. + Run the test using the following command form: + + $ bitbake image -c testsdk + + + + + + testsdk_ext: + + These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. + The code for these tests resides in meta/lib/oeqa/esdk. + To run the tests, use the following command form: + + $ bitbake image -c testsdkext + + + + + + + + oe-build-perf-test: + + These tests run through commonly used usage scenarios and measure the performance times. + The code for these tests resides in meta/lib/oeqa/buildperf. + To run the tests, use the following command form: + + $ oe-build-perf-test options + The + command takes a number of options, such as where to place the + test results. The Autobuilder Helper Scripts include the + build-perf-test-wrapper script with + examples of how to use the oe-build-perf-test from the command + line. + Use the oe-git-archive command to store + test results into a Git repository. + Use the oe-build-perf-report command to + generate text reports and HTML reports with graphs of the + performance data. For examples, see http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html + and http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt. + + The tests are contained in + lib/oeqa/buildperf/test_basic.py. + + + + + + + + + +
+ +
+ Test Examples + + This section provides example tests for each of the tests listed in the How Tests Map to Areas of Code section. + For oeqa tests, testcases for each area reside in the main test directory at + meta/lib/oeqa/selftest/cases directory. + For oe-selftest. bitbake testcases reside in the lib/bb/tests/ + directory. + +
+ <filename>bitbake-selftest</filename> + + A simple test example from lib/bb/tests/data.py is: + + class DataExpansions(unittest.TestCase): + def setUp(self): + self.d = bb.data.init() + self.d["foo"] = "value_of_foo" + self.d["bar"] = "value_of_bar" + self.d["value_of_foo"] = "value_of_'value_of_foo'" + + def test_one_var(self): + val = self.d.expand("${foo}") + self.assertEqual(str(val), "value_of_foo") + + + In this example, a DataExpansions class + of tests is created, derived from standard python unittest. The class has a common + setUp function which is shared by all the tests in the + class. A simple test is then added to test that when a variable is expanded, the + correct value is found. + Bitbake selftests are straightforward python unittest. Refer to the Python + unittest documentation for additional information on writing these tests at: https://docs.python.org/3/library/unittest.html. +
+ +
+ <filename>oe-selftest</filename> + + These tests are more complex due to the setup required behind the scenes for full + builds. Rather than directly using Python's unittest, the code wraps most of the + standard objects. The tests can be simple, such as testing a command from within the + OE build environment using the following + example: + class BitbakeLayers(OESelftestTestCase): + def test_bitbakelayers_showcrossdepends(self): + result = runCmd('bitbake-layers show-cross-depends') + self.assertTrue('aspell' in result.output, msg = "No dependencies + were shown. bitbake-layers show-cross-depends output: + %s"% result.output) + + This example, taken from + meta/lib/oeqa/selftest/cases/bblayers.py, creates a + testcase from the OESelftestTestCase + class, derived from unittest.TestCase, which runs the + bitbake-layers command and checks the output to ensure it + contains something we know should be here. + The oeqa.utils.commands module contains Helpers which can + assist with common tasks, including: + + Obtaining the value of a bitbake variable: Use + oeqa.utils.commands.get_bb_var() or use + oeqa.utils.commands.get_bb_vars() for more than + one variable + + + Running a bitbake invocation for a build: Use + oeqa.utils.commands.bitbake() + + + Running a command: Use + oeqa.utils.commandsrunCmd() + + + There is also a oeqa.utils.commands.runqemu() function for + launching the runqemu command for testing things within a + running, virtualized image. + You can run these tests in parallel. Parallelism works per test class, so tests + within a given test class should always run in the same build, while tests in + different classes or modules may be split into different builds. There is no data + store available for these tests since the tests launch the + bitbake command and exist outside of its context. As a + result, common bitbake library functions (bb.*) are also unavailable. +
+ +
+ <filename>testimage</filename> + + These tests are run once an image is up and running, either on target hardware or + under QEMU. As a result, they are assumed to be running in a target image + environment, as opposed to a host build environment. A simple example from + meta/lib/oeqa/runtime/cases/python.py contains the + following: + class PythonTest(OERuntimeTestCase): + @OETestDepends(['ssh.SSHTest.test_ssh']) + @OEHasPackage(['python3-core']) + def test_python3(self): + cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, + jbeyq', 'rot13'))\"" + status, output = self.target.run(cmd) + msg = 'Exit status was not 0. Output: %s' % output + self.assertEqual(status, 0, msg=msg) + + In this example, the OERuntimeTestCase + class wraps unittest.TestCase. Within the test, + self.target represents the target system, where commands + can be run on it using the run() method. + To ensure certain test or package dependencies are met, you can use the + OETestDepends and OEHasPackage + decorators. For example, the test in this example would only make sense if + python3-core is installed in the image. +
+ +
+ <filename>testsdk_ext</filename> + + These tests are run against built extensible SDKs (eSDKs). The tests can assume + that the eSDK environment has already been setup. An example from + meta/lib/oeqa/sdk/cases/devtool.py contains the + following: + class DevtoolTest(OESDKExtTestCase): + @classmethod + def setUpClass(cls): + myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) + + @classmethod + def tearDownClass(cls): + shutil.rmtree(cls.myapp_dst) + def _test_devtool_build(self, directory): + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') + def test_devtool_build_make(self): + self._test_devtool_build(self.myapp_dst) + In + this example, the devtool command is tested to see whether a + sample application can be built with the devtool build command + within the eSDK. +
+ +
+ <filename>testsdk</filename> + + These tests are run against built SDKs. The tests can assume that an SDK has + already been extracted and its environment file has been sourced. A simple example + from meta/lib/oeqa/sdk/cases/python2.py contains the + following: + class Python3Test(OESDKTestCase): + def setUp(self): + if not (self.tc.hasHostPackage("nativesdk-python3-core") or + self.tc.hasHostPackage("python3-core-native")): + raise unittest.SkipTest("No python3 package in the SDK") + + def test_python3(self): + cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" + output = self._run(cmd) + self.assertEqual(output, "Hello, world\n") + In + this example, if nativesdk-python3-core has been installed into the SDK, the code + runs the python3 interpreter with a basic command to check it is working correctly. + The test would only run if python3 is installed in the SDK. +
+ +
+ <filename>oe-build-perf-test</filename> + + The performance tests usually measure how long operations take and the resource + utilisation as that happens. An example from + meta/lib/oeqa/buildperf/test_basic.py contains the + following: + class Test3(BuildPerfTestCase): + + def test3(self): + """Bitbake parsing (bitbake -p)""" + # Drop all caches and parse + self.rm_cache() + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', + 'bitbake -p (no caches)') + # Drop tmp/cache + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', + 'bitbake -p (no tmp/cache)') + # Parse with fully cached data + self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', + 'bitbake -p (cached)') + This + example shows how three specific parsing timings are measured, with and without + various caches, to show how BitBake’s parsing performance trends over time. +
+
+
+ Considerations When Writing Tests + When writing good tests, there are several things to keep in mind. Since things + running on the Autobuilder are accessed concurrently by multiple workers, consider the + following: + + Running "cleanall" is not permitted + This can delete files from DL_DIR which would potentially break other builds + running in parallel. If this is required, DL_DIR must be set to an isolated + directory. + + + Running "cleansstate" is not permitted + This can delete files from SSTATE_DIR which would potentially break other builds + running in parallel. If this is required, SSTATE_DIR must be set to an isolated + directory. Alternatively, you can use the "-f" option with the + bitbake command to "taint" tasks by changing the sstate + checksums to ensure sstate cache items will not be reused. + + + Tests should not change the metadata + This is particularly true for oe-selftests since these can run in parallel and + changing metadata leads to changing checksums, which confuses BitBake while running + in parallel. If this is necessary, copy layers to a temporary location and modify + them. Some tests need to change metadata, such as the devtool tests. To prevent the + metadate from changes, set up temporary copies of that data first. + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual-style.css b/documentation/test-manual/test-manual-style.css new file mode 100644 index 0000000000..15ff718b16 --- /dev/null +++ b/documentation/test-manual/test-manual-style.css @@ -0,0 +1,989 @@ +/* + Generic XHTML / DocBook XHTML CSS Stylesheet. + + Browser wrangling and typographic design by + Oyvind Kolas / pippin@gimp.org + + Customised for Poky by + Matthew Allum / mallum@o-hand.com + + Thanks to: + Liam R. E. Quin + William Skaggs + Jakub Steiner + + Structure + --------- + + The stylesheet is divided into the following sections: + + Positioning + Margins, paddings, width, font-size, clearing. + Decorations + Borders, style + Colors + Colors + Graphics + Graphical backgrounds + Nasty IE tweaks + Workarounds needed to make it work in internet explorer, + currently makes the stylesheet non validating, but up until + this point it is validating. + Mozilla extensions + Transparency for footer + Rounded corners on boxes + +*/ + + + /*************** / + / Positioning / +/ ***************/ + +body { + font-family: Verdana, Sans, sans-serif; + + min-width: 640px; + width: 80%; + margin: 0em auto; + padding: 2em 5em 5em 5em; + color: #333; +} + +h1,h2,h3,h4,h5,h6,h7 { + font-family: Arial, Sans; + color: #00557D; + clear: both; +} + +h1 { + font-size: 2em; + text-align: left; + padding: 0em 0em 0em 0em; + margin: 2em 0em 0em 0em; +} + +h2.subtitle { + margin: 0.10em 0em 3.0em 0em; + padding: 0em 0em 0em 0em; + font-size: 1.8em; + padding-left: 20%; + font-weight: normal; + font-style: italic; +} + +h2 { + margin: 2em 0em 0.66em 0em; + padding: 0.5em 0em 0em 0em; + font-size: 1.5em; + font-weight: bold; +} + +h3.subtitle { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; + font-size: 142.14%; + text-align: right; +} + +h3 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 140%; + font-weight: bold; +} + +h4 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 120%; + font-weight: bold; +} + +h5 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +h6 { + margin: 1em 0em 0em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +.authorgroup { + background-color: transparent; + background-repeat: no-repeat; + padding-top: 256px; + background-image: url("figures/test-manual-title.png"); + background-position: left top; + margin-top: -256px; + padding-right: 50px; + margin-left: 0px; + text-align: right; + width: 740px; +} + +h3.author { + margin: 0em 0me 0em 0em; + padding: 0em 0em 0em 0em; + font-weight: normal; + font-size: 100%; + color: #333; + clear: both; +} + +.author tt.email { + font-size: 66%; +} + +.titlepage hr { + width: 0em; + clear: both; +} + +.revhistory { + padding-top: 2em; + clear: both; +} + +.toc, +.list-of-tables, +.list-of-examples, +.list-of-figures { + padding: 1.33em 0em 2.5em 0em; + color: #00557D; +} + +.toc p, +.list-of-tables p, +.list-of-figures p, +.list-of-examples p { + padding: 0em 0em 0em 0em; + padding: 0em 0em 0.3em; + margin: 1.5em 0em 0em 0em; +} + +.toc p b, +.list-of-tables p b, +.list-of-figures p b, +.list-of-examples p b{ + font-size: 100.0%; + font-weight: bold; +} + +.toc dl, +.list-of-tables dl, +.list-of-figures dl, +.list-of-examples dl { + margin: 0em 0em 0.5em 0em; + padding: 0em 0em 0em 0em; +} + +.toc dt { + margin: 0em 0em 0em 0em; + padding: 0em 0em 0em 0em; +} + +.toc dd { + margin: 0em 0em 0em 2.6em; + padding: 0em 0em 0em 0em; +} + +div.glossary dl, +div.variablelist dl { +} + +.glossary dl dt, +.variablelist dl dt, +.variablelist dl dt span.term { + font-weight: normal; + width: 20em; + text-align: right; +} + +.variablelist dl dt { + margin-top: 0.5em; +} + +.glossary dl dd, +.variablelist dl dd { + margin-top: 0em; + margin-left: 25.5em; +} + +.glossary dd p, +.variablelist dd p { + margin-top: 0em; + margin-bottom: 1em; +} + + +div.calloutlist table td { + padding: 0em 0em 0em 0em; + margin: 0em 0em 0em 0em; +} + +div.calloutlist table td p { + margin-top: 0em; + margin-bottom: 1em; +} + +div p.copyright { + text-align: left; +} + +div.legalnotice p.legalnotice-title { + margin-bottom: 0em; +} + +p { + line-height: 1.5em; + margin-top: 0em; + +} + +dl { + padding-top: 0em; +} + +hr { + border: solid 1px; +} + + +.mediaobject, +.mediaobjectco { + text-align: center; +} + +img { + border: none; +} + +ul { + padding: 0em 0em 0em 1.5em; +} + +ul li { + padding: 0em 0em 0em 0em; +} + +ul li p { + text-align: left; +} + +table { + width :100%; +} + +th { + padding: 0.25em; + text-align: left; + font-weight: normal; + vertical-align: top; +} + +td { + padding: 0.25em; + vertical-align: top; +} + +p a[id] { + margin: 0px; + padding: 0px; + display: inline; + background-image: none; +} + +a { + text-decoration: underline; + color: #444; +} + +pre { + overflow: auto; +} + +a:hover { + text-decoration: underline; + /*font-weight: bold;*/ +} + +/* This style defines how the permalink character + appears by itself and when hovered over with + the mouse. */ + +[alt='Permalink'] { color: #eee; } +[alt='Permalink']:hover { color: black; } + + +div.informalfigure, +div.informalexample, +div.informaltable, +div.figure, +div.table, +div.example { + margin: 1em 0em; + padding: 1em; + page-break-inside: avoid; +} + + +div.informalfigure p.title b, +div.informalexample p.title b, +div.informaltable p.title b, +div.figure p.title b, +div.example p.title b, +div.table p.title b{ + padding-top: 0em; + margin-top: 0em; + font-size: 100%; + font-weight: normal; +} + +.mediaobject .caption, +.mediaobject .caption p { + text-align: center; + font-size: 80%; + padding-top: 0.5em; + padding-bottom: 0.5em; +} + +.epigraph { + padding-left: 55%; + margin-bottom: 1em; +} + +.epigraph p { + text-align: left; +} + +.epigraph .quote { + font-style: italic; +} +.epigraph .attribution { + font-style: normal; + text-align: right; +} + +span.application { + font-style: italic; +} + +.programlisting { + font-family: monospace; + font-size: 80%; + white-space: pre; + margin: 1.33em 0em; + padding: 1.33em; +} + +.tip, +.warning, +.caution, +.note { + margin-top: 1em; + margin-bottom: 1em; + +} + +/* force full width of table within div */ +.tip table, +.warning table, +.caution table, +.note table { + border: none; + width: 100%; +} + + +.tip table th, +.warning table th, +.caution table th, +.note table th { + padding: 0.8em 0.0em 0.0em 0.0em; + margin : 0em 0em 0em 0em; +} + +.tip p, +.warning p, +.caution p, +.note p { + margin-top: 0.5em; + margin-bottom: 0.5em; + padding-right: 1em; + text-align: left; +} + +.acronym { + text-transform: uppercase; +} + +b.keycap, +.keycap { + padding: 0.09em 0.3em; + margin: 0em; +} + +.itemizedlist li { + clear: none; +} + +.filename { + font-size: medium; + font-family: Courier, monospace; +} + + +div.navheader, div.heading{ + position: absolute; + left: 0em; + top: 0em; + width: 100%; + background-color: #cdf; + width: 100%; +} + +div.navfooter, div.footing{ + position: fixed; + left: 0em; + bottom: 0em; + background-color: #eee; + width: 100%; +} + + +div.navheader td, +div.navfooter td { + font-size: 66%; +} + +div.navheader table th { + /*font-family: Georgia, Times, serif;*/ + /*font-size: x-large;*/ + font-size: 80%; +} + +div.navheader table { + border-left: 0em; + border-right: 0em; + border-top: 0em; + width: 100%; +} + +div.navfooter table { + border-left: 0em; + border-right: 0em; + border-bottom: 0em; + width: 100%; +} + +div.navheader table td a, +div.navfooter table td a { + color: #777; + text-decoration: none; +} + +/* normal text in the footer */ +div.navfooter table td { + color: black; +} + +div.navheader table td a:visited, +div.navfooter table td a:visited { + color: #444; +} + + +/* links in header and footer */ +div.navheader table td a:hover, +div.navfooter table td a:hover { + text-decoration: underline; + background-color: transparent; + color: #33a; +} + +div.navheader hr, +div.navfooter hr { + display: none; +} + + +.qandaset tr.question td p { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; +} + +.qandaset tr.answer td p { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; +} +.answer td { + padding-bottom: 1.5em; +} + +.emphasis { + font-weight: bold; +} + + + /************* / + / decorations / +/ *************/ + +.titlepage { +} + +.part .title { +} + +.subtitle { + border: none; +} + +/* +h1 { + border: none; +} + +h2 { + border-top: solid 0.2em; + border-bottom: solid 0.06em; +} + +h3 { + border-top: 0em; + border-bottom: solid 0.06em; +} + +h4 { + border: 0em; + border-bottom: solid 0.06em; +} + +h5 { + border: 0em; +} +*/ + +.programlisting { + border: solid 1px; +} + +div.figure, +div.table, +div.informalfigure, +div.informaltable, +div.informalexample, +div.example { + border: 1px solid; +} + + + +.tip, +.warning, +.caution, +.note { + border: 1px solid; +} + +.tip table th, +.warning table th, +.caution table th, +.note table th { + border-bottom: 1px solid; +} + +.question td { + border-top: 1px solid black; +} + +.answer { +} + + +b.keycap, +.keycap { + border: 1px solid; +} + + +div.navheader, div.heading{ + border-bottom: 1px solid; +} + + +div.navfooter, div.footing{ + border-top: 1px solid; +} + + /********* / + / colors / +/ *********/ + +body { + color: #333; + background: white; +} + +a { + background: transparent; +} + +a:hover { + background-color: #dedede; +} + + +h1, +h2, +h3, +h4, +h5, +h6, +h7, +h8 { + background-color: transparent; +} + +hr { + border-color: #aaa; +} + + +.tip, .warning, .caution, .note { + border-color: #fff; +} + + +.tip table th, +.warning table th, +.caution table th, +.note table th { + border-bottom-color: #fff; +} + + +.warning { + background-color: #f0f0f2; +} + +.caution { + background-color: #f0f0f2; +} + +.tip { + background-color: #f0f0f2; +} + +.note { + background-color: #f0f0f2; +} + +.glossary dl dt, +.variablelist dl dt, +.variablelist dl dt span.term { + color: #044; +} + +div.figure, +div.table, +div.example, +div.informalfigure, +div.informaltable, +div.informalexample { + border-color: #aaa; +} + +pre.programlisting { + color: black; + background-color: #fff; + border-color: #aaa; + border-width: 2px; +} + +.guimenu, +.guilabel, +.guimenuitem { + background-color: #eee; +} + + +b.keycap, +.keycap { + background-color: #eee; + border-color: #999; +} + + +div.navheader { + border-color: black; +} + + +div.navfooter { + border-color: black; +} + +.writernotes { + color: red; +} + + + /*********** / + / graphics / +/ ***********/ + +/* +body { + background-image: url("images/body_bg.jpg"); + background-attachment: fixed; +} + +.navheader, +.note, +.tip { + background-image: url("images/note_bg.jpg"); + background-attachment: fixed; +} + +.warning, +.caution { + background-image: url("images/warning_bg.jpg"); + background-attachment: fixed; +} + +.figure, +.informalfigure, +.example, +.informalexample, +.table, +.informaltable { + background-image: url("images/figure_bg.jpg"); + background-attachment: fixed; +} + +*/ +h1, +h2, +h3, +h4, +h5, +h6, +h7{ +} + +/* +Example of how to stick an image as part of the title. + +div.article .titlepage .title +{ + background-image: url("figures/white-on-black.png"); + background-position: center; + background-repeat: repeat-x; +} +*/ + +div.preface .titlepage .title, +div.colophon .title, +div.chapter .titlepage .title, +div.article .titlepage .title +{ +} + +div.section div.section .titlepage .title, +div.sect2 .titlepage .title { + background: none; +} + + +h1.title { + background-color: transparent; + background-image: url("figures/test-title.png"); + background-repeat: no-repeat; + height: 256px; + text-indent: -9000px; + overflow:hidden; +} + +h2.subtitle { + background-color: transparent; + text-indent: -9000px; + overflow:hidden; + width: 0px; + display: none; +} + + /*************************************** / + / pippin.gimp.org specific alterations / +/ ***************************************/ + +/* +div.heading, div.navheader { + color: #777; + font-size: 80%; + padding: 0; + margin: 0; + text-align: left; + position: absolute; + top: 0px; + left: 0px; + width: 100%; + height: 50px; + background: url('/gfx/heading_bg.png') transparent; + background-repeat: repeat-x; + background-attachment: fixed; + border: none; +} + +div.heading a { + color: #444; +} + +div.footing, div.navfooter { + border: none; + color: #ddd; + font-size: 80%; + text-align:right; + + width: 100%; + padding-top: 10px; + position: absolute; + bottom: 0px; + left: 0px; + + background: url('/gfx/footing_bg.png') transparent; +} +*/ + + + + /****************** / + / nasty ie tweaks / +/ ******************/ + +/* +div.heading, div.navheader { + width:expression(document.body.clientWidth + "px"); +} + +div.footing, div.navfooter { + width:expression(document.body.clientWidth + "px"); + margin-left:expression("-5em"); +} +body { + padding:expression("4em 5em 0em 5em"); +} +*/ + + /**************************************** / + / mozilla vendor specific css extensions / +/ ****************************************/ +/* +div.navfooter, div.footing{ + -moz-opacity: 0.8em; +} + +div.figure, +div.table, +div.informalfigure, +div.informaltable, +div.informalexample, +div.example, +.tip, +.warning, +.caution, +.note { + -moz-border-radius: 0.5em; +} + +b.keycap, +.keycap { + -moz-border-radius: 0.3em; +} +*/ + +table tr td table tr td { + display: none; +} + + +hr { + display: none; +} + +table { + border: 0em; +} + + .photo { + float: right; + margin-left: 1.5em; + margin-bottom: 1.5em; + margin-top: 0em; + max-width: 17em; + border: 1px solid gray; + padding: 3px; + background: white; +} + .seperator { + padding-top: 2em; + clear: both; + } + + #validators { + margin-top: 5em; + text-align: right; + color: #777; + } + @media print { + body { + font-size: 8pt; + } + .noprint { + display: none; + } + } + + +.tip, +.note { + background: #f0f0f2; + color: #333; + padding: 20px; + margin: 20px; +} + +.tip h3, +.note h3 { + padding: 0em; + margin: 0em; + font-size: 2em; + font-weight: bold; + color: #333; +} + +.tip a, +.note a { + color: #333; + text-decoration: underline; +} + +.footnote { + font-size: small; + color: #333; +} + +/* Changes the announcement text */ +.tip h3, +.warning h3, +.caution h3, +.note h3 { + font-size:large; + color: #00557D; +} diff --git a/documentation/test-manual/test-manual-test-process.xml b/documentation/test-manual/test-manual-test-process.xml new file mode 100644 index 0000000000..0b5036cd2c --- /dev/null +++ b/documentation/test-manual/test-manual-test-process.xml @@ -0,0 +1,109 @@ + %poky; ] > + + + +Project Testing and Release Process +
+ Day to Day Development + + This section details how the project tests changes, through automation on the + Autobuilder or with the assistance of QA teams, through to making releases. + + The project aims to test changes against our test matrix before those changes are + merged into the master branch. As such, changes are queued up in batches either in the + master-next branch in the main trees, or in user trees such as + ross/mut in poky-contrib (Ross Burton + helps review and test patches and this is his testing tree). + We have two broad categories of test builds, including "full" and "quick". On the + Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in + the UI. Use our Autobuilder console view to see where me manage most test-related items, + available at: https://autobuilder.yoctoproject.org/typhoon/#/console. + Builds are triggered manually when the test branches are ready. The builds are + monitored by the SWAT team. For additional information, see https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team. If + successful, the changes would usually be merged to the master + branch. If not successful, someone would respond to the changes on the mailing list + explaining that there was a failure in testing. The choice of quick or full would depend + on the type of changes and the speed with which the result was required. + The Autobuilder does build the master branch once daily for + several reasons, in particular, to ensure the current master branch + does build, but also to keep yocto-testresults (http://git.yoctoproject.org/cgit.cgi/yocto-testresults/), buildhistory + (http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/), + and our sstate up to date. On the weekend, there is a master-next build instead to + ensure the test results are updated for the less frequently run targets. + Performance builds (buildperf-* targets in the console) are triggered separately every + six hours and automatically push their results to the buildstats repository at: http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/. + The 'quick' targets have been selected to be the ones which catch the most failures or + give the most valuable data. We run 'fast' ptests in this case for example but not the + ones which take a long time. The quick target doesn't include *-lsb builds for all + architectures, some world builds and doesn't trigger performance tests or ltp testing. + The full build includes all these things and is slower but more comprehensive. +
+ +
+ Release Builds + + The project typically has two major releases a year with a six month cadence in April + and October. Between these there would be a number of milestone releases (usually four) + with the final one being stablization only along with point releases of our stable + branches. + The build and release process for these project releases is similar to that in Day to Day Development, in that the a-full target + of the Autobuilder is used but in addition the form is configured to generate and + publish artefacts and the milestone number, version, release candidate number and other + information is entered. The box to "generate an email to QA"is also checked. + When the build completes, an email is sent out using the send-qa-email script in the + yocto-autobuilder-helper repository to the list of people + configured for that release. Release builds are placed into a directory in https://autobuilder.yocto.io/pub/releases on the Autobuilder which + is included in the email. The process from here is more manual and control is + effectively passed to release engineering. The next steps include: + + QA teams respond to the email saying which tests they plan to run and when + the results will be available. + + + QA teams run their tests and share their results in the yocto- + testresults-contrib repository, along with a summary of their findings. + + + + Release engineering prepare the release as per their process. + + + Test results from the QA teams are included into the release in separate + directories and also uploaded to the yocto-testresults repository alongside + the other test results for the given revision. + + + The QA report in the final release is regenerated using resulttool to + include the new test results and the test summaries from the teams (as + headers to the generated report). + + + The release is checked against the release checklist and release readiness + criteria. + + + A final decision on whether to release is made by the YP TSC who have + final oversight on release readiness. + + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual-understand-autobuilder.xml b/documentation/test-manual/test-manual-understand-autobuilder.xml new file mode 100644 index 0000000000..7541305350 --- /dev/null +++ b/documentation/test-manual/test-manual-understand-autobuilder.xml @@ -0,0 +1,312 @@ + %poky; ] > + + + +Understanding the Yocto Project Autobuilder +
+ Execution Flow within the Autobuilder + The “a-full” and “a-quick” targets are the usual entry points into the Autobuilder and + it makes sense to follow the process through the system starting there. This is best + visualised from the Autobuilder Console view (https://autobuilder.yoctoproject.org/typhoon/#/console). + Each item along the top of that view represents some “target build” and these targets + are all run in parallel. The ‘full’ build will trigger the majority of them, the “quick” + build will trigger some subset of them. The Autobuilder effectively runs whichever + configuration is defined for each of those targets on a seperate buildbot worker. To + understand the configuration, you need to look at the entry on + config.json file within the + yocto-autobuilder-helper repository. The targets are defined in + the ‘overrides’ section, a quick example could be qemux86-64 which looks + like: + "qemux86-64" : { + "MACHINE" : "qemux86-64", + "TEMPLATE" : "arch-qemu", + "step1" : { + "extravars" : [ + "IMAGE_FSTYPES_append = ' wic wic.bmap'" + ] + } + }, + And + to expand that, you need the “arch-qemu” entry from the “templates” section, which looks + like: + "arch-qemu" : { + "BUILDINFO" : true, + "BUILDHISTORY" : true, + "step1" : { + "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", + "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" + }, + "step2" : { + "SDKMACHINE" : "x86_64", + "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", + "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" + }, + "step3" : { + "BUILDHISTORY" : false, + "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], + "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] + } + }, + Combining + these two entries you can see that “qemux86-64” is a three step build where the “bitbake + BBTARGETS” would be run, then “bitbake SANITYTARGETS” for each step; all for + MACHINE=”qemx86-64” but with differing SDKMACHINE settings. In step 1 an extra variable + is added to the auto.conf file to enable wic image + generation. + While not every detail of this is covered here, you can see how the templating + mechanism allows quite complex configurations to be built up yet allows duplication and + repetition to be kept to a minimum. + The different build targets are designed to allow for parallelisation, so different + machines are usually built in parallel, operations using the same machine and metadata + are built sequentially, with the aim of trying to optimise build efficiency as much as + possible. + The config.json file is processed by the scripts in the Helper + repository in the scripts directory. The following section details + how this works. +
+ +
+ Autobuilder Target Execution Overview + + For each given target in a build, the Autobuilder executes several steps. These are + configured in yocto-autobuilder2/builders.py and roughly consist + of: + + Run clobberdir + This cleans out any previous build. Old builds are left around to allow + easier debugging of failed builds. For additional information, see clobberdir. + + + Obtain yocto-autobuilder-helper + This step clones the yocto-autobuilder-helper git + repository. This is necessary to prevent the requirement to maintain all the + release or project-specific code within Buildbot. The branch chosen matches + the release being built so we can support older releases and still make + changes in newer ones. + + + Write layerinfo.json + This transfers data in the Buildbot UI when the build was configured to + the Helper. + + + Call scripts/shared-repo-unpack + This is a call into the Helper scripts to set up a checkout of all the + pieces this build might need. It might clone the BitBake repository and the + OpenEmbedded-Core repository. It may clone the Poky repository, as well as + additional layers. It will use the data from the + layerinfo.json file to help understand the + configuration. It will also use a local cache of repositories to speed up + the clone checkouts. For additional information, see Autobuilder Clone + Cache. + This step has two possible modes of operation. If the build is part of a + parent build, its possible that all the repositories needed may already be + available, ready in a pre-prepared directory. An "a-quick" or "a-full" build + would prepare this before starting the other sub-target builds. This is done + for two reasons: + + the upstream may change during a build, for example, from a + forced push and this ensures we have matching content for the + whole build + + + if 15 Workers all tried to pull the same data from the same + repos, we can hit resource limits on upstream servers as they + can think they are under some kind of network attack + + This pre-prepared directory is shared among the Workers over + NFS. If the build is an individual build and there is no "shared" directory + available, it would clone from the cache and the upstreams as necessary. + This is considered the fallback mode. + + + Call scripts/run-config + This is another call into the Helper scripts where its expected that the + main functionality of this target will be executed. + + +
+
+ Autobuilder Technology + The Autobuilder has Yocto Project-specific functionality to allow builds to operate + with increased efficiency and speed. +
+ clobberdir + When deleting files, the Autobuilder uses clobberdir, which + is a special script that moves files to a special location, rather than deleting + them. Files in this location are deleted by an rm command, + which is run under ionice -c 3. For example, the deletion only + happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor + runs this deletion. See Autobuilder + Worker Janitor. +
+
+ Autobuilder Clone Cache + Cloning repositories from scratch each time they are required was slow on the + Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on + the Workers. Data is fetched from these during clones first, then "topped up" with + later revisions from any upstream when necesary. The cache is maintained by the + Autobuilder Worker Janitor. See Autobuilder Worker Janitor. +
+
+ Autobuilder Worker Janitor + This is a process running on each Worker that performs two basic operations, + including background file deletion at IO idle (see Target Execution: clobberdir) and + maintainenance of a cache of cloned repositories to improve the speed the system can + checkout repositories. +
+
+ Shared DL_DIR + The Workers are all connected over NFS which allows DL_DIR to be shared between + them. This reduces network accesses from the system and allows the build to be sped + up. Usage of the directory within the build system is designed to be able to be + shared over NFS. +
+
+ Shared SSTATE_DIR + The Workers are all connected over NFS which allows the + sstate directory to be shared between them. This means once + a Worker has built an artefact, all the others can benefit from it. Usage of the + directory within the directory is designed for sharing over NFS. +
+
+ Resulttool + All of the different tests run as part of the build generate output into + testresults.json files. This allows us to determine which + tests ran in a given build and their status. Additional information, such as failure + logs or the time taken to run the tests, may also be included. + Resulttool is part of OpenEmbedded-Core and is used to manipulate these json + results files. It has the ability to merge files together, display reports of the + test results and compare different result files. + For details, see https://wiki.yoctoproject.org/wiki/Resulttool. +
+
+
+ run-config Target Execution + The scripts/run-config execution is where most of the work within + the Autobuilder happens. It runs through a number of steps; the first are general setup + steps that are run once and include: + + Set up any buildtools-tarball if configured. + + + Call "buildhistory-init" if buildhistory is configured. + + + For each step that is configured in config.json, it will perform + the following: + + ## WRITER's question: What does "logging in as stepXa" and others refer to + below? ## + + + Add any layers that are specified using the + bitbake-layers add-layer command (logging as + stepXa) + + + Call the scripts/setup-config script to + generate the necessary auto.conf configuration file for + the build + + + Run the bitbake BBTARGETS command (logging + as stepXb) + + + Run the bitbake SANITYTARGETS command + (logging as stepXc) + + + Run the EXTRACMDS command, which are run + within the BitBake build environment (logging as stepXd) + + + Run the EXTRAPLAINCMDS command(s), which + are run outside the BitBake build environment (logging as stepXd) + + + Remove any layers added in step 1 using the + bitbake-layers remove-layer command (logging as + stepXa) + + + + Once the execution steps above complete, run-config executes a + set of post-build steps, including: + + Call scripts/publish-artifacts to collect + any output which is to be saved from the build. + + + Call scripts/collect-results to collect any + test results to be saved from the build. + + + Call scripts/upload-error-reports to send + any error reports generated to the remote server. + + + Cleanup the build directory using clobberdir if the + build was successful, else rename it to “build-renamed” for potential future + debugging. + + +
+
+ Deploying Yocto Autobuilder + The most up to date information about how to setup and deploy your own Autbuilder can + be found in README.md in the yocto-autobuilder2 repository. + We hope that people can use the yocto-autobuilder2 code directly + but it is inevitable that users will end up needing to heavily customise the + yocto-autobuilder-helper repository, particularly the + config.json file as they will want to define their own test + matrix. + The Autobuilder supports wo customization options: + + variable substitution + + + overlaying configuration files + + The standard config.json minimally attempts to allow + substitution of the paths. The Helper script repository includes a + local-example.json file to show how you could override these + from a separate configuration file. Pass the following into the environment of the + Autobuilder: + $ ABHELPER_JSON="config.json local-example.json" + As + another example, you could also pass the following into the + environment: + $ ABHELPER_JSON="config.json /some/location/local.json" + One + issue users often run into is validation of the config.json files. + A tip for minimizing issues from invalid json files is to use a Git + pre-commit-hook.sh script to verify the JSON file before + committing it. Create a symbolic link as + follows: + $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual.xml b/documentation/test-manual/test-manual.xml new file mode 100755 index 0000000000..9d3c0354de --- /dev/null +++ b/documentation/test-manual/test-manual.xml @@ -0,0 +1,103 @@ + %poky; ] > + + + + + + + + + + + + Yocto Project Test Environment Manual + + + + + + &ORGNAME; + + &ORGEMAIL; + + + + + + 3.1.1 + TBD + DRAFT - Work-in-Progress - posted June 16, 2020 + + + + + ©RIGHT_YEAR; + Linux Foundation + + + + + Permission is granted to copy, distribute and/or modify this document under + the terms of the + Creative Commons Attribution-Share Alike 2.0 UK: England & Wales as published by + Creative Commons. + + Manual Notes + + + This version of the + Yocto Project Test Environment Manual + is for the &YOCTO_DOC_VERSION; release of the + Yocto Project. + To be sure you have the latest version of the manual + for this release, go to the + Yocto Project documentation page + and select the manual from that site. + Manuals from the site are more up-to-date than manuals + derived from the Yocto Project released TAR files. + + + If you located this manual through a web search, the + version of the manual might not be the one you want + (e.g. the search might have returned a manual much + older than the Yocto Project version with which you + are working). + You can see all Yocto Project major releases by + visiting the + Releases + page. + If you need a version of this manual for a different + Yocto Project release, visit the + Yocto Project documentation page + and select the manual set by using the + "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE" + pull-down menus. + + + To report any inaccuracies or problems with this + manual, send an email to the Yocto Project + discussion group at + yocto@yoctoproject.com or log into + the freenode #yocto channel. + + + + + + + + + + + + +