Testing Phases / Parts What are their meanings

First let me start of by saying the QA is a very methodical occupation, there is a time for every little step, with that being said we can now discuss the different stages in the version testing cycles:

Functionality - this phase involves checking the a new added functionality has indeed been added for starters and further more we would test and make sure that it is functioning as expected according to system requirements.

Regression - in this phase the tester will continue on to test that old / previous system capabilities have not been affected by the change code done for the new features.

Bug Fixes - this phase is not part of every version cycle tests since it depends whether any bugs were fixed, basically QA is expected to go over the version Release Notes and verify that all the bugs marked by the programmer as fixed in this version have indeed been fixed. e.g. QA will have to simulate the conditions which brought on the defeact in the privious version and very that the defect does not occur in the new version.

Performance - every product has certain dimensioning requirements, regarding the amount of input it can handle, in this phase the QA will verify that the system can indeed handle the high marker for system input as specified in the System requirements by the product manager.

Stability - in the same manner where a certain maximum input load is defined for a specific system then a certain steady state should also be decided upon between product manager and system engineer.
according to this steady state stability tests are run checking system functionality over an extended period of time to see no system error occurs during the said time, stability tests can run from hours to days or weeks

Attacks - in different SUT's (Systems Under Test) this test phase may be called in different names, in general this test phase will include abnormal use of the system to check for ability to receive abuse by hostile/ untrained operator.

Sanity - this phase is one of the critical phases. basically it is a collection of several test cases from all of the above phases (almost) which is meant to give a quick picture of system state after receiving a bug fix /patch.
Again this will not test the entire system but rather a brief run through the system in order to reflect system state to relevant parties

Cyclic Testing vs. Itertive Testing

Iterative testing -

Simply means testing that is repeated, or iterated, multiple times. Iterative usability testing matters because the ultimate goal of all usability work is to improve usability.

Iterative testing is usually helpful in early development steps and can be considered as part of the integration process.

Cyclic Tests -

This is the type of testing we usually associate with classical testing in Cyclic testing the work is more methodical.

The content of a released version is usually divided into several parts.

We would usually try to test new version features in the earlier cycles later moving on to regression.

We should incorporate bug fixes cycles during this cycles provided we receive bug fixes during the test phases.

We will discuss the different parts in more details in future blogs.

Bug / Problem reporting the right and the wrong way to do it

There is an entire Methodology behind bug/ problem reporting.
There are many tools used for bug tracking and reporting.

all of these do not matter, we need to keep in mind the purpose and the goal for which we report a bug, if we do this then all will be well.
According to the Bugzilla site :(https://developer.mozilla.org/en/Bug_writing_guidelines)
The guidelines to problem reporting are detailed below:

• Be precise
• Be clear - explain it so others can reproduce the bug
• One bug per report
• No bug is too trivial to report - small bugs may hide big bugs
• Clearly separate fact from speculation
• Reproduce your bug using a latest version of the software, to see whether it has already been fixed.
• When system configuration is relevant specify it.
• When log files are available provide them.

Following these guidelines will ensure the programmer understands you and has all the information to solve the problem... it obviously does not guarantee bug fix times

Trouble Shooting Guide lines

During the earlier days of my career I found my self constantly being called on Costumer site for trouble shooting reasons .

I would usually discover when arriving there that the so called complex problem is easily solved by a simple and short analysis session.

I tried to lay out some guidelines for others to follow in order to enable them to do the same.

What it basically comes down to is asking the right set of questions in the right order.

the questions which guided me were:

1. locate the first step/ component to indicate a problem
2. what is the cpu/ mem utilization?
3. which application is consuming the most resources?
4. open the log files
   
5. look for the most recent error message
   
6. what does it say
   
7. what does it mean
   

When answering these last two questions you can generally solve most problems.
At one time I traveled to one of our costumer sites in the east.
Problem report was system was malfunctioning and missing several input responses.
When arriving on site I noticed that the gate component was running at high CPU utilization (~90% steady state)
when reviewing the logs I noticed a configuration error.
Once I handled this I noticed that the CPU did not drop.
I reviewed the logs again and noticed that the same error message is being printed at a very high rate ( which was causing the high CPU utilization).
I turned to the programmer and requested to reduce the debug level for the specific message (since it was pointing to an not mapped input type and not a system problem)
Once this was accomplished all other problems were gone.
Problem Solved.

If the field Engineer /Support follows this rules he can generally get to the bottom of the problems or at least point R&D in the right direction

This in my opinion is the true meaning of On site Field Engineering, other wise it would mean that the Field Engineer is expected only to perform the "Next, Next, Next, ..." installation type.... Is this what Field Engineers are for ?!?

I Don't think so.

QA On Costumer Site - rules and guide lines

There are very specific criteria to determine when QA is needed on site , as I see it only if these criteria are met then there is a point to assigning QA to the costumer site.

1. Costumer has specific system inputs which can not be simulated in a lab.
2. Specific problems can not be simulated in the lab .
   
3. Tests did not complete prior to deployment, and costumer is willing to provide a testing environment.
   

In general as you can see I think there are very few occasions justifying QA on site , how ever since on many occasions system know how is lacking and QA is forced to travel to site . When On site the following rules should be met...

1. QA has traveled on site for a specific purpose it does not include costumer instructions/ politics on site installations etc...
2. when reporting from site QA should maintain the same guidelines directing him when reporting a bug from the lab.
1. be clear and to the point
   
2. describe a specific and exact scenario
3. provide all possible available debug information
4. describe system configuration with all necessary detail

Once these rules and guidelines are followed we are certain to make the most of the QA visit on site.

When this need actually arises

QA and IT - to install or not to install - that is the question

When dealing with QA yo must remember that there are extensive IT requirements, this is of course product dependent , but for true QA to exist, several things must occur which relate to IT.
First and foremost Product Manager/ System manager must define :

• on which Platform the Product must run.
• What are the minimal system requirements for the product.

once the above has been determined it is extremely vital that the QA performs tests for all of the System configuration defined;

And here comes the tricky part who is in charge of the installation and maintenance of these environments.
In the organization in which I am part of w do not have a separate IT group. this creates havoc in my opinion for the following reason.

1. Not all QA are predefined to have IT capabilities and thus a lot of time is wasted due to insufficient skills.
2. Since no proper IT exists there are no predefined sets of instructions as to how and what exactly to set up with in the Platform itself
3. QA time is "wasted" on OS setups.
4. Since QA are not exactly qualified for this they find it difficult to issue clear instructions to system Field Engineers on how to set up costumer systems, thus creating in consistency of OS types and configuration ... which basically will lead to unexplained problems from specific costumer with out any clear reason.
5. No one can handle new hardware certification/ old hardware End-Of-Life replacement.
   

In my opinion it is vital to have a strong IT group in every company , and not just for the sake of handling the tedious tasks of system setup but in order to set up guidelines so that we create a consistency between all of the systems installed among the different costumers

QA Types

There are several types of QA which common today in the industry.
Each type of QA obviously comes in order to answer a specific need.
It is important to identify what are needs and not expect wrong things of the wrong types of QA

Unit Tester - The unit tester in general is one of the first to perform QA on the product , this type of QA will usually be done with in the development team, prior to releasing any version, this is basically a preliminary verification that the component functions as expected to a controlled (simulated) set of inputs and produces the expected outputs (results).
========================================================================
Black box / Functionality testing - This type of testing does not require the tester to have any familiarity with the architecture/technology of the tested component, he generally performs a predefined set of actions (inputs) and expects an appropriate response (output) in turn. This is usually the kind of tester people associate with testing , some one going over a graphical interface, checking all the check boxes and different screens. obviously the skills required for such QA are not high, unfortunately some think this type reflect on all QA.
========================================================================
System/ E2E testing - This type of QA usually involves high level QA ... there may not be drill down to a specific functionality but in this type of QA you most certainly have to have a good familiarity with the system flow and its architecture, further to this you must be very well versed in the technology relevant to this product ;e.g., for a telephony system you will have to have telephony protocol familiarity.
========================================================================
White Box Testing - This type of QA depends on the QA's familiarity with the technology in question in order to be able to challenge the application/product, in some respect the type of person required for this type of QA is similar to the E2E tester, how ever this QA will perform drill down testing on the product.

Product Life Cycle

In order to better understand the role of the QA with in the entire process of product delivery we first need to understand the process as a whole.

In the beginning there was the Marketing Product Manager; this person in-fact is suppose to outline the product road map and list down the requirements of the product.
The product manager will usually base his requirement lists according to market trends and need which he was exposed to, thus laying out a road map for the product and its list of features, further input/ specifications may come from specific costumers which have shown an interest in the product it self.

Next comes the System Engineer - his role will be to translate the sometimes vague requests of the Product Manager into R&D / development tasks.
basically the system engineer will lay out the method in which the solution is to be designed... his role is to design a system which answers the current requirements but keeps in mind future needs scalability and much more .
you may read more about Sys. Engineering in the following blog :
http://design-to-last.com/

The System engineering role is vitals since development with out a plan equals in my mind to a chicken running around with out a head.

Once the System engineer has completed his role come in the developers and take the plans laid out by the System engineer and devise the actual code structure .

Once code structure and design has be done it is time to get down to the dirty part of writing the actual code.

It is a common misconception to think that the QA role will follow only at the end after the programmer has completed his work.

THIS IS WRONG!!!!

If we remember the tasks we listed in the previous segment we shall see that the QA is suppose to accompany the product throughout the life cycle from infancy to maturity.

Good QA will be made part even in the early stages so they are at least aware of the Product manager list of requirements so see that they actually match the requirements from the costumer.

then , it is up to the QA to make sure the above mentioned list correlates and is fully covered in the design layed out by the system engineer.

last but not least QA will then test the product delivered by the programmer and make sure it complies with the list of requirements listed above and further to this that it functions properly with out affecting the environment surrounding it, be it the machine running the product or other machines in the network.

Good QA will try to predict costumer requirements and methods of use for the product and then test that these are actually possible .

QA Tasks

As mentioned in the previous segment the QA has several tasks, let us write them down:

1. Make sure a project is completed based upon previously agreed specifications, standards and functionality requirements.
2. Make sure a project is completed with out defect and possible problems.
3. Monitors and try to improve the development process from the beginning.
4. Preventive orientation.
   

After listing those formidable tasks we can see that the QA role is a vital one in the process of supplying a product to a costumer.

In the following segments we will discuss in further details these tasks.

What is QA

I thought that since my blog will be dealing with QA and my view on its role , I'd better spend some time in defining "what is QA?" at least in my opinion.

Quality Assurance makes sure the project will be completed based on the previously agreed specifications, standards and functionality required without defects and possible problems. It monitors and tries to improve the development process from the beginning of the project to ensure this. It is oriented to "prevention". (http://www.asknumbers.com/QualityAssuranceandTesting.aspx)

Let us take a look at this "dictionary" definition for QA, and try to gather from it the tasks for the QA engineer, let us also try and understand what are the inputs needed by the QA engineer in order to be able accomplish his tasks

A Little about Why I decided to write this Blog

I chose to write this Blog for several reasons:
1. As I mentioned in my account details I have been in the Telecom market for quite some time now, while I may be considered a new comer in respect to QA I thought other may gain some things from my experience.
2. I have certain views in respect to QA role and job description, which don't always align with company views.
3. At certain points i get frustrated with the establishment and thought this would be a good way to vent some of this frustration.