A digital camera was released in the market and after sometime a customer came back complaining that .......
Problem :The camera's software was crashing continously while he tried to capture images.
FIR : They first asked some basic information from the customer and made an attempt to reproduce the issue but it was not reproducible.
Analysis :
1) Software is crashing - so they listed out all possible scenarios where the software could crash.
2) Tried putting the camera to all listed possibilities.
3) Giving a report to the management that it was unable to be reproduced.
Further Analysis :
Taking this as a challenge , a great tester had the following concern -
1) He was concerened what festival was it ?
2) He was concerned what was the customer was trying to capture ?
3) He was concerned what is the frequency of the crash the customer noticed ?
4) He was concerened about all other surrounding factors like temperuature , place where the festival was celebrated ... etc ?
Outcome of Analysis :
1) The festival was Diwali or Deepavali !
2) The customer was trying to capture fire crackers , explosion , fireworks display in an open ground at around 22 00 hours ( night ).
What a fantastic tester was he !
He identified the problem and told the developers
"The camera software is crashing when the viewfinder is black ( because of dark night ) and a sudden gush of light through the firecracker explosion is putting the DSP ( digital signal processor ) to a load/stress beyond its boundaries to display the image. And at this point of time when the user tries to capture it simply crashes"
What he said ?
was right !
Finally they fixed it and now today its a robust digital camera !
____ End of a probable true story _____
Summary of lessons -
1) Collect real time data for testing a product/application.
2) When you are unable to reproduce an issue , think out of the box , think out of the black box.
3) When you are unable to reproduce an issue call someone who can dig deeper than you.
4) Dont assign test case draft to the person who is going to test it, he may omit complicated cases to bring down his/her complexity in executing it.
5) Review the cases with a non team member/customers/domain specailist/tester network.
6) When you release a product : aniticpate and be prepared to face such situation.
7) Hate the product while testing and love the product after its release - emotional testing ( ha new concept introduced by me , i do it , to be frank )
8) Good Testers not only report bugs , they suggest a solution ! ( more to come abt this )
9) Keep learning to be a good tester !
10) Test all your products in India , its Testing Bowl of the world !
11) Don't terminate an issue just because you are unable to reproduce , it could kill your company !
Saturday, March 3, 2007
Software Testing basics
Software testing is the process used to help identify the correctness, completeness, security and quality of developed computer software.
The IEEE definition for testing is:
“The process of exercising or evaluating a system by manual or automatic means to verify that it satisfies specified requirements or to identify differences between expected and actual results”.
The definition of Myers says,
“Testing is the process of executing a program with the intent of finding errors”.
Testing is an activity aimed at evaluating an attribute or capability of a program or system to determine that it meets its required results. It is a process of evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.
Testing is used to determine the status of the product during and after the build or do component of the process. The role of testing changes as the type of process used to build the product changes.
What is the goal of testing?
The main goal of testing is to check if the system meets the user requirements and check for the systems reliability.
Why software testing?
To produce quality product.
To reduce the failure cost and maintenance cost.
The IEEE definition for testing is:
“The process of exercising or evaluating a system by manual or automatic means to verify that it satisfies specified requirements or to identify differences between expected and actual results”.
The definition of Myers says,
“Testing is the process of executing a program with the intent of finding errors”.
Testing is an activity aimed at evaluating an attribute or capability of a program or system to determine that it meets its required results. It is a process of evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results.
Testing is used to determine the status of the product during and after the build or do component of the process. The role of testing changes as the type of process used to build the product changes.
What is the goal of testing?
The main goal of testing is to check if the system meets the user requirements and check for the systems reliability.
Why software testing?
To produce quality product.
To reduce the failure cost and maintenance cost.
SDLC
The software development life cycle (SDLC) is the entire process of formal, logical steps taken to develop a software product. Software development is the process of developing software through successive phases in an orderly way. This process includes not only the actual writing of code but also the preparation of requirements and objectives, the design of what is to be coded, and confirmation that what is developed has met objectives.
The software development life cycle (SDLC) is a conceptual model used in project management that describes the stages involved in an information system development project, from an initial feasibility study through maintenance of the completed application.
Typical phases of software development:
1) Requirement Analysis
2) Software design
3) Development or Coding
4) Testing
5) Maintenance
Requirement Analysis:
The new system requirements are defined. The requirements of a desired software product are extracted. Based the business scenario the SRS (Software Requirement Specification) document is prepared in this phase. The purpose of this document is to specify the functional requirements of the software that will be produced by the SOS project group. The specifications are intended to guide the group through the development process.
Design:
Plans are laid out concerning the physical construction, hardware, operating systems, programming, communications, and security issues for the software. Design phase is concerned with making sure the software system will meet the requirements of the product. It should also ensure that the future requirements will also be addressed.
Development (or) Coding:
Programs are developed by the developers with the help of the design. The design is reduced to code by the software engineers.
Testing:
Testing is evaluating the software to check for the user requirements. Here the software is evaluated with intent of finding defects.
Maintenance:
Once the new system is up and running for awhile, it should be exhaustively evaluated. Maintenance must be kept up rigorously at all times. Users of the system should be kept up-to-date concerning the latest modifications and procedures.
Software development life cycle models:
The waterfall model:
This is the best known and oldest model, with a linear and sequential method that has goals for each development phase. In waterfall model there are no iterative or overlapping steps. They state requirements, analyze them, design a solution approach, develop code, test, deploy, and maintain. After each step is finished, the process proceeds to the next step. One drawback of the waterfall is that it is static about user requirements. It does not allow for much revision.
Rapid application development (RAD):
The problem with previous methodology was that applications took so long to build that requirements had changed before the system was complete, often resulting in unusable systems. This model is based on the concept that better products can be developed more quickly by: using groups to gather system requirements, prototyping and reiterative testing of designs and less formality of team communications such as reviews.
Advantages:
Increased speed of development and increased quality.
Disadvantages:
Reduced Scalability and reduced features.
The prototyping model: In this model, a prototype is built, tested, and then reworked as necessary until an acceptable prototype is finally achieved from which the complete system or product can now be developed.
The spiral model: This model of development combines the features of the prototyping model and the waterfall model. Each phase starts with a design goal and ends with the client (who may be internal) reviewing the progress thus far. Analysis and engineering efforts are applied at each phase of the project, with an eye toward the end goal of the project. The spiral model is favored for large, expensive, and complicated projects.
The software development life cycle (SDLC) is a conceptual model used in project management that describes the stages involved in an information system development project, from an initial feasibility study through maintenance of the completed application.
Typical phases of software development:
1) Requirement Analysis
2) Software design
3) Development or Coding
4) Testing
5) Maintenance
Requirement Analysis:
The new system requirements are defined. The requirements of a desired software product are extracted. Based the business scenario the SRS (Software Requirement Specification) document is prepared in this phase. The purpose of this document is to specify the functional requirements of the software that will be produced by the SOS project group. The specifications are intended to guide the group through the development process.
Design:
Plans are laid out concerning the physical construction, hardware, operating systems, programming, communications, and security issues for the software. Design phase is concerned with making sure the software system will meet the requirements of the product. It should also ensure that the future requirements will also be addressed.
Development (or) Coding:
Programs are developed by the developers with the help of the design. The design is reduced to code by the software engineers.
Testing:
Testing is evaluating the software to check for the user requirements. Here the software is evaluated with intent of finding defects.
Maintenance:
Once the new system is up and running for awhile, it should be exhaustively evaluated. Maintenance must be kept up rigorously at all times. Users of the system should be kept up-to-date concerning the latest modifications and procedures.
Software development life cycle models:
The waterfall model:
This is the best known and oldest model, with a linear and sequential method that has goals for each development phase. In waterfall model there are no iterative or overlapping steps. They state requirements, analyze them, design a solution approach, develop code, test, deploy, and maintain. After each step is finished, the process proceeds to the next step. One drawback of the waterfall is that it is static about user requirements. It does not allow for much revision.
Rapid application development (RAD):
The problem with previous methodology was that applications took so long to build that requirements had changed before the system was complete, often resulting in unusable systems. This model is based on the concept that better products can be developed more quickly by: using groups to gather system requirements, prototyping and reiterative testing of designs and less formality of team communications such as reviews.
Advantages:
Increased speed of development and increased quality.
Disadvantages:
Reduced Scalability and reduced features.
The prototyping model: In this model, a prototype is built, tested, and then reworked as necessary until an acceptable prototype is finally achieved from which the complete system or product can now be developed.
The spiral model: This model of development combines the features of the prototyping model and the waterfall model. Each phase starts with a design goal and ends with the client (who may be internal) reviewing the progress thus far. Analysis and engineering efforts are applied at each phase of the project, with an eye toward the end goal of the project. The spiral model is favored for large, expensive, and complicated projects.
Regression Testing
Regression testing is testing the module in which a bug was identified earlier along with the impacted areas to ensure that this fix has not introduced any further defects.
The purpose of regression testing is to ensure that previously detected and fixed issues really are fixed, they do not reappear, and new issues are not introduced into the program as a result of the changes made to fix the issues.
Regression testing also referred to as verification testing, is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. It is a quality control measure to ensure that the newly modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.
Regression Testing is in general a black box testing strategy where test case execution of previously written test cases, that has exposed bugs, is done to check whether previously fixed faults have reemerged. In a test suite, all the tests that has caused bug are written and are re-tested whenever changes are made to the program to fix any bug. But this is a tedious process as after every compilation it is difficult to go through the process of retesting all the test cases repeatedly. To make this process simpler regression testing is automated using some testing tools.
Typically regression testing should be performed on a daily basis. Once an issue in the defect tracking database has been fixed it is reassigned back for final resolution. Now it can be either reopens the issue, if it has not been satisfactorily addressed, or close the issue if it has, indeed, been fixed.
The purpose of regression testing is to ensure that previously detected and fixed issues really are fixed, they do not reappear, and new issues are not introduced into the program as a result of the changes made to fix the issues.
Regression testing also referred to as verification testing, is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. It is a quality control measure to ensure that the newly modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.
Regression Testing is in general a black box testing strategy where test case execution of previously written test cases, that has exposed bugs, is done to check whether previously fixed faults have reemerged. In a test suite, all the tests that has caused bug are written and are re-tested whenever changes are made to the program to fix any bug. But this is a tedious process as after every compilation it is difficult to go through the process of retesting all the test cases repeatedly. To make this process simpler regression testing is automated using some testing tools.
Typically regression testing should be performed on a daily basis. Once an issue in the defect tracking database has been fixed it is reassigned back for final resolution. Now it can be either reopens the issue, if it has not been satisfactorily addressed, or close the issue if it has, indeed, been fixed.
Testing Lifecycle
In a software development process, there are five different phases. They are
- Requirement analysis
- Design
- Development (or) Coding
- Testing
- Maintenance.
Here the testing comes in fourth phase. But the actual testing process begins during the first phase itself i.e. testing process begins during the Requirement analysis phase itself.
The steps in the testing process are as follows.
1. Requirement analysis
Testing should begin in the requirements phase of the software life cycle (SDLC). The actual requirement should be understand clearly with the help of Requirement Specification documents, Functional Specification documents, Design Specification documents, Use case Documents etc.
During the requirement analysis the following should be considered.-Are the definitions and descriptions of the required capabilities precise? -Is there clear delineation between the system and its environment?-Can the requirements be realized in practice?-Can the requirements be tested effectively?
2. Test Planning
During this phase Test Strategy, Test Plan, Test Bed will be created.A test plan is a systematic approach in testing a system or software. The plan should identify:-Which aspects of the system should be tested?-Criteria for success.-The methods and techniques to be used.-Personnel responsible for the testing.-Different Test phase and Test Methodologies-Manual and Automation Testing-Defect Mgmt, Configuration Mgmt, Risk Mgmt. Etc-Evaluation & identification – Test, Defect tracking tools
3. Test Environment
SetupDuring this phase the required environment will be setup will be done. The following should also be taken in account.- Network connectivity’s-All the Software/ tools Installation and configuration-Coordination with Vendors and others.
4. Test Case Design
During this phase-Test Scenarios will be identified.-Test Cases will be prepared.-Test data and Test scripts prepared.-Test case reviews will be conducted.
5. Test Automation
In this phase the requirement for the automation will be identified. The tools that are to be used will be identified. Designing framework, scripting, script integration, Review and approval will be undertaken in this phase.
6. Test Execution and Defect Tracking
Testers execute the software based on the plans and tests and report any errors found to the development team. In this phase -Test cases will be executed.-Test Scripts will be tested.-Test Results will be analyzed.-Raised the defects and tracking for its closure.
7. Test Reports
Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
-Test summary reports will be prepared-Test Metrics and process Improvements made-Build release-Receiving acceptance
- Requirement analysis
- Design
- Development (or) Coding
- Testing
- Maintenance.
Here the testing comes in fourth phase. But the actual testing process begins during the first phase itself i.e. testing process begins during the Requirement analysis phase itself.
The steps in the testing process are as follows.
1. Requirement analysis
Testing should begin in the requirements phase of the software life cycle (SDLC). The actual requirement should be understand clearly with the help of Requirement Specification documents, Functional Specification documents, Design Specification documents, Use case Documents etc.
During the requirement analysis the following should be considered.-Are the definitions and descriptions of the required capabilities precise? -Is there clear delineation between the system and its environment?-Can the requirements be realized in practice?-Can the requirements be tested effectively?
2. Test Planning
During this phase Test Strategy, Test Plan, Test Bed will be created.A test plan is a systematic approach in testing a system or software. The plan should identify:-Which aspects of the system should be tested?-Criteria for success.-The methods and techniques to be used.-Personnel responsible for the testing.-Different Test phase and Test Methodologies-Manual and Automation Testing-Defect Mgmt, Configuration Mgmt, Risk Mgmt. Etc-Evaluation & identification – Test, Defect tracking tools
3. Test Environment
SetupDuring this phase the required environment will be setup will be done. The following should also be taken in account.- Network connectivity’s-All the Software/ tools Installation and configuration-Coordination with Vendors and others.
4. Test Case Design
During this phase-Test Scenarios will be identified.-Test Cases will be prepared.-Test data and Test scripts prepared.-Test case reviews will be conducted.
5. Test Automation
In this phase the requirement for the automation will be identified. The tools that are to be used will be identified. Designing framework, scripting, script integration, Review and approval will be undertaken in this phase.
6. Test Execution and Defect Tracking
Testers execute the software based on the plans and tests and report any errors found to the development team. In this phase -Test cases will be executed.-Test Scripts will be tested.-Test Results will be analyzed.-Raised the defects and tracking for its closure.
7. Test Reports
Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
-Test summary reports will be prepared-Test Metrics and process Improvements made-Build release-Receiving acceptance
Subscribe to:
Comments (Atom)