|
 This is a new era in Build and Release management. Continuous integration brings more frequent builds. More frequent builds means more frequent updates for testers, developers, and other stakeholders. This brings out new use cases / user stories: 1. As a Test engineer, I want to know, is the "Product Build output" valid to take forward for testing. 2. As a Developer, I want to know, did the product build go through without any compilation issues? Was the Integration done correctly? 3. As a stakeholder (Product owner, Program manager), I want to know, what is the status of product build?
Many build management tools (e.g. Hudson) provide the methods for notifying task completion via e-mails and they also provide a dashboard that shows task status. But "All green" status does not always mean that the product build is "good" and can be promoted to the testing phase. This article is about creating effective "Build Announcements".
Let's understand the user stories,
What does tester want to know?
Typically, a software testing environment is a controlled (separate) environment from the Build environment. So when the product needs to be tested, Testers will promote the product to the testing environment (which may be in a different subnet, with different firewall settings) and it needs to be installed (older versions may need to uninstalled before installing latest build) and then, finally, it is ready to test.
Clearly the above steps involve efforts (which may raise questions), What if installer itself is broken? What if the component failed to compile? Testers will find out anyway but then it is too late. The testing environment needs to be rolled back (Uninstall the latest build and install the previous build) in order to continue with the testing.
What does the developer want to know?
Software developers want to know that the build went through without compilation issues and components are integrated properly. Just parsing (e.g. looking at error patters) compiler logs is not good enough because software products usually depend on many run time binaries and integration issues will be found only while using (testing) the software product.
Software developers depend upon the testing team to know the major integration issues or they themselves need to do the minimal testing to ensure that the software build is properly integrated.
What do all the stakeholders want to know?
Stakeholders (e.g. Product owners, program managers etc...) want to know the status of the product build in a rather "Boolean" way (Pass or Fail) because the build management tool's dashboard will not show the clear status. For example, compilation task may show green but the run time component integration may fail.
What about automated task completion notification?
Build management tools supports automated task notification mails. It just notifies us that a particular build / task has been completed. It is also for a particular task not for a complete product build.
The level of continuous integration differs from product to product. Less "weighty" products may build the whole product (including packaging) every time the source code changes. Heavy weight products (which can takes 6 to 8 hrs to build the complete product package) still depends on nightly builds for full build and it might build a few components for every source code change.
Either way, once the complete build is over, it is available for testing and then someone/something needs to ensure the quality of build.
So how do we satisfy all the above requirements? How do we make sure that everyone (Tester, developer and stakeholders) gets the required information.
Build Announcement
Here we need the process of "Build Announcement". This is the mail notification that contains the "Boolean" result of build status (Pass/Fail).
How do we assess that the build has passed/failed? - It should be based on the "Quality Gate", You can name it as a BAT (Build Acceptance Test) or Sanity Test quality gate.
I prefer the "Sanity test" because it verifies the overall sanity of product which involves some part of functionality testing.
Sanity Testing
Sanity Test quality gate should involve the mandatory test cases which will help to ensure the build's status.
For example, A software product can have the following test cases to ensure the product is ready to take forward to testing,
1. Compiler logs - Should not have compiler issues
2. Packaging logs - Should not have packaging issues
3. Product installation - Product should install with out issues
4. Product invocation - Product should come up without crash
5. Product de installation - Product should un install with out issue
Hey, Wait! Many test cases also involve testing functions, Should I test the product build before announce the build?
Yes!, Here comes the Joy of Automation. Automate your Sanity test cases and Build announcement.
Sanity Test Automation
In Nokia, We have automated both sanity testing and Build announcements. Sanity test frame work architecture looks like this,
As the architecture diagram explains, Once the product build is completed, the Build server invokes the Sanity test cases via the Sanity test framework and the build will be tested (You can add some minimal no GUI test cases also) for sanity.
If all the mandatory test cases are passed, the build will be shared in a production build share area and the build will be announced as successful build - else the build will be shared in a temp build share and it will be announced as "failure" build.
A minimal sanity testing helps here to announce the builds.
Here is our success build announcement format,
Sub: Build Announcement: Sprint__
-- build
Sanity Test Report
-----------------------------------
1. Download_Build: PASS *
2. Unzip: PASS *
3. Test_compile_log: PASS *
No.of Compilation Warnings: 42636
4. Test_CMworkarea_log: FAIL
No.of Explicit Conflicts: 2
5. Test_Install: PASS *
6. Test_Invokation: PASS *
7. Test_Deinstall: PASS *
8. Cleanup_Build: PASS
Passed : 7
Failed : 1
Total : 8
Product used for this test :
.zip
Note: * indicates mandatory TestCase.
Build PASSED Sanity Test.
Build Location: \\\
Build Report: \\\\Build_Report.html
Build Log Location: \\\\Build_Log
Sanity Test Log Location: \\\\
.log
*** AUTO GENERATED MAIL - Pls do not Reply***
SDK Sanity Test Automation
The above test cases help us to determine the success or failure build.
How "Build Announcement" is useful?
Since the Build announcement and sanity testing is automated, it will happen over the night and everyone gets the build announcement before step in to office.
For Testers: Successful Build Announcement makes sure that latest build can be taken forward for testing. So they can take it into test machines without worrying about the roll back and efforts spend to move to latest builds will not go waste.
For Developer: The test cases include analyzing log files and invoking the product. If both are fine, we can make sure that Compilation & Integration is successful. Developer will get to know the Integration status without waiting for testing results.
For Stakeholder: Build announcement is a Boolean type. It says, build is passed or failed. No need to look in to the dashboard and interpret multiple tasks status. It also helps to collects the metrics, for example how many broken builds in this particular sprint.
If software product has a complete automated test environment, Sanity testing and build announcement will not add value for testers but still few test cases (Ex: install, deinstall) can be moved as part of sanity test and Build Announcement will be useful to others (Developers and Stakeholders).
Conclusion
Build announcement with a sanity test quality gate tests the builds even before it is announced (released) for consumption. It will improve quality because it finds issues as early as possible. It is not just analyzing the compiler log files = but it actually tests the product and it will be helpful to find out integration issues.
More test cases can be added for improving the build success confidence level. Since the test cases are automated, it avoids the human errors and improves the quality.
Build Announcements helps to reduce the engineering effort and improve quality!
Bakkiaraj M is currently working as Build Engineer in Nokia. He started his career as Build Engineer in Hewlett Packard. For the past 4 years, he has been in Build and Release management. He is fond of build / test automation. He wrote perl tools for automated patching, Sanity test, Build management, Build metrics collection etc… He has been working with almost all major OS architecture builds including Windows, HPUX, AIX and Linux. He has built the software using Msbuild, make, gmake, Tiny::Make and Ant. He is familiar with Build management tools like cruise control, Hudson and SCM tools CM Synergy, Clearcase and Subversion. He likes scripting; He uses Perl, Python and UNIX shells. He doez familiar with Installshield packaging tool. He has a master's degree in “Software Engineering” from Bharathiyar University, Tamilnadu, India.
Trackback(0)
Comments 
Write comment
 |
TrackBack URI for this entry
Subscribe to this comment's feed