Thursday 8 December 2011

Commands in UNIX

Basic UNIX Commands and there description.

cat --- for creating and displaying short files
Syntax: cat filename
Ex: cat > filename
cat file1 >> file2

chmod --- change permissions for three classes of users: the user (u), group (g), or others (o) by entering
Syntax: chmod ugo filename

cd --- change directory
Syntax: cd directory
EX: From the subdirectory, /users/jones/memos, change to the directory, /users/jones/samples with the command,
cd /users/jones/samples

cp --- for copying files
Syntax: cp [options] old_filename new_filename
Ex: cp file1 file2

ftp --- connect to a remote machine to download or upload files
Syntax: ftp options hostname
options
-d enable debugging.
-g disable filename globbing.
-i turn off interactive prompts.
-v verbose on. show all responses from remote server.
ftp hostname by default will connect you to the system, you must have a login id to be able to transfer the files. Two types of files can be transferred, ASCII or Binary. bin at ftp> prompt will set the transfer to binary. Practice FTP by ftping to nic.funet.fi loggin in as anomymous with password being your e-mail address.

head --- display first 10 lines of a file.
Syntax: head [-count] filename
Ex: If you want first 50 lines you can use head -50 filename or for 37 lines head -37 filename.

ls --- see what files you have
Examples: ls List files in current directory
ls -l Lists files in “long” format

lpr --- standard print command (see also print )
Syntax: [ lpr -P(<--capital P) printername ]

more --- use to read files
Syntax: more filename
More is a command used to read text files. For example, we could do this:
% more poems
The effect of this to let you read the file "poems ". It probably will not fit in one screen, so you need to know how to "turn pages". Here are the basic commands:
q --- quit more
spacebar --- read next page
return key --- read next line
b --- go back one page
For still more information, use the command man more.


mkdir --- create directory
Syntax: [ mkdir directoryname ]

mv --- for moving and renaming files
Syntax:[ mv filename directory OR mv filename newfilename ]

pwd --- find out what directory you are in (will print your home directory on screen).
Syntax: [ pwd ]

rm --- remove a file
Syntax: [ rm filename ]

rmdir --- remove directory
Options:
rm -r directory_name will remove all files even if directory is not empty.
rmdir sandeep is how you use it to remove sandeep directory.
rmdir -p will remove directories and any parent directories that are empty.
rmdir -s will suppress standard error messages caused by -p.

sort --- sort file
Syntax: sort filename
Example : suppose we have a file dict with contents
red rojo
green verde
blue azul
white blanco
black negro
Then we can do this:
% sort dict
black negro
blue azul
green verde
red rojo
white blanco
Here the output of sort went to the screen. To store the output in file we do this:
% sort dict >dict.sorted
You can check the contents of the file dict.sorted using cat , more , or emacs .
tail --- display last 10 lines of a file.
Syntax: tail [-count] filename
Ex: last 50 lines then use ----- tail -50 filename.

telnet --- log in to another machine( remote system)
Syntax:[ telnet hostname ]

wc --- count characters, words, lines in a file depending upon the option.
Syntax: wc [options] filename
Options
wc -l filename will print total number of lines in a file.
wc -w filename will print total number of words in a file.
wc -c filename will print total number of characters in a file.



tar --- create an archive, add or extract files
Use create compressed archives of directories and files, and also to extract directories and files from an archive. Example:
% tar -tvzf foo.tar.gz
displays the file names in the compressed archive foo.tar.gz while
% tar -xvzf foo.tar.gz
extracts the files.


rsh --- remote shell
Use this command if you want to work on a computer different from the one you are currently working on. One reason to do this is that the remote machine might be faster. For example, the command
% rsh solitude
connects you to the machine solitude. This is one of our public workstations and is fairly fast.


setenv --- set an environment variable
Ex:   % echo $PRINTER
labprinter
% setenv PRINTER myprinter
% echo $PRINTER
myprinter


ncftp --- especially good for downloading files via anonymous ftp.
Use ncftp for anonymous ftp --- that means you don't have to have a password.
% ncftp ftp.fubar.net
Connected to ftp.fubar.net
> get jokes.txt
The file jokes.txt is downloaded from the machine ftp.fubar.net.

print --- custom print command (see also lpr )
This is a moderately intelligent print command.
% print foo
% print notes.ps
% print manuscript.dvi
In each case print does the right thing, regardless of whether the file is a text file (like foo ), a postcript file (like notes.ps, or a dvi file (like manuscript.dvi. In these examples the file is printed on the default printer. To see what this is, do
% print
and read the message displayed. To print on a specific printer, do this:
% print foo jwb321
% print notes.ps jwb321
% print manuscript.dvi jwb321
To change the default printer, do this:
% setenv PRINTER jwb321

grep(Global search for Regular Expression and Print.) --- search file
Use this command to search for information in a file or files. For example, suppose that we have a file dict whose contents are
red rojo
green verde
blue azul
white blanco
black negro
Then we can look up items in our file like this;
% grep red dict
red rojo
% grep blanco dict
white blanco
% grep brown dict
%
Notice that no output was returned by grep brown. This is because "brown" is not in our dictionary file.
Grep can also be combined with other commands. For example, if one had a file of phone numbers named "ph", one entry per line, then the following command would give an alphabetical list of all persons whose name contains the string "Fred".
% grep Fred ph | sort
Alpha, Fred: 333-6565
Beta, Freddie: 656-0099
Frederickson, Molly: 444-0981
Gamma, Fred-George: 111-7676
Zeta, Frederick: 431-0987
The symbol "|" is called "pipe." It pipes the output of the grep command into the input of the sort command.
For more information on grep, consult
% man grep


date --- display date (Use this command to check the date and time.)
Syntax : % date
Fri Jan 6 08:52:42 MST 1995

echo --- echo argument
Ex: $ echo This is a test.

UNIX for Software Testers

WHAT IS UNIX?

UNIX is an operating system which was first developed in the 1960s, and has been under constant development ever since. By operating system, we mean the suite of programs which make the computer work. It is a stable, multi-user, multi-tasking system for servers, desktops and laptops.
UNIX systems also have a graphical user interface (GUI) similar to Microsoft Windows which provides an easy to use environment. However, knowledge of UNIX is required for operations which aren't covered by a graphical program, or for when there is no windows interface available, for example, in a telnet session.

TYPES OF UNIX
There are many different versions of UNIX, although they share common similarities. The most popular varieties of UNIX are Sun Solaris, GNU/Linux, and MacOS X.


THE UNIX OPERATING SYSTEM

The UNIX operating system is made up of three parts; the kernel, the shell and the programs.

The kernel

The kernel of UNIX is the hub of the operating system: it allocates time and memory to programs and handles the filestore and communications in response to system calls.
As an illustration of the way that the shell and the kernel work together, suppose a user types rm myfile (which has the effect of removing the file myfile). The shell searches the filestore for the file containing the program rm, and then requests the kernel, through system calls, to execute the program rm on myfile. When the process rm myfile has finished running, the shell then returns the UNIX prompt % to the user, indicating that it is waiting for further commands.

The shell


The shell acts as an interface between the user and the kernel. When a user logs in, the login program checks the username and password, and then starts another program called the shell. The shell is a command line interpreter (CLI). It interprets the commands the user types in and arranges for them to be carried out. The commands are themselves programs: when they terminate, the shell gives the user another prompt (% on our systems).
The adept user can customise his/her own shell, and users can use different shells on the same machine. Staff and students in the school have the tcsh shell by default.
The tcsh shell has certain features to help the user inputting commands.
Filename Completion - By typing part of the name of a command, filename or directory and pressing the [Tab] key, the tcsh shell will complete the rest of the name automatically. If the shell finds more than one name beginning with those letters you have typed, it will beep, prompting you to type a few more letters before pressing the tab key again.
History - The shell keeps a list of the commands you have typed in. If you need to repeat a command, use the cursor keys to scroll up and down the list or type history for a list of previous commands.

FILES AND PROCESSES

Everything in UNIX is either a file or a process.
A process is an executing program identified by a unique PID (process identifier).
A file is a collection of data. They are created by users using text editors, running compilers etc.
Examples of files:
a document (report, essay etc.)
the text of a program written in some high-level programming language
instructions comprehensible directly to the machine and incomprehensible to a casual user, for example, a collection of binary digits (an executable or binary file);
a directory, containing information about its contents, which may be a mixture of other directories (subdirectories) and ordinary files.


THE DIRECTORY STRUCTURE

All the files are grouped together in the directory structure. The file-system is arranged in a hierarchical structure, like an inverted tree. The top of the hierarchy is traditionally called root (written as a slash / )



Commands in UNIX

Monday 28 November 2011

Test Case For Captcha?

1. Ensure that the CAPTCHA code screen should be viewable.
2. Ensure that the CAPTCHA code should be in Image format.
3. Ensure that the CAPTCHA code should be reusable.(able to get new code by clicked link)
4. Ensure that the CAPTCHA code should generate new code once the page is refreshed.
5. Ensure that the CAPTCHA code should generate new code once entered the wrong code.
6. Ensure that the CAPTCHA code should not able to copy/paste.
7. Ensure that the CAPTCHA code should be Case Sensitive or vice versa(According to the Requirement).
8. Ensure that the CAPTCHA code should not distruot the other fields in the form once entered wrong.(Other form fields should not clear)
9. Application should not accept the Invalid CAPTCHA code on form feed and should display proper alert message.
10. Application should accept the Valid CAPTCHA code on form feed.

Testcase for Yahoo Mail Page after login

Steps:

Testcase1: To verify that when we click mail button whether it list all the compose and check mail etc options or not
Description: click on the mail button
Expected result: Clicking of the mail button lists all the options check mail and compose etc

Testcase2: To verify that when we click check mail option in the mail list whether it takes you to inbox page or not
Description: click on check mail option
Expected result: check mail option opens the inbox page

Testcase3: To verify that when you click the inbox whether it displays u r received mails or not
Description: click the inbox button
Expected result: It lists all the mails u received in the inbox

Testcase4: To verify when u click the compose option in the mail button whether it takes u to compose page where u can compose and send mails
Description: click on the compose option in the mail button
Expected result: it takes u to compose page

Testcase5: To verify that after writing message when you click on ‘Send’ whether mail is sent to the address where you specified
Description: give mail id for which you want to send the message in the ‘To’ field and write the message in compose box and click on send button
Expected result: it sending the mail to the mail id which u are given in the TO field

Testcase6: To verify if you give wrong id whether it gives failure notice or not
Description: Give wrong mail id in compose page in the ‘To’ field and see what happens
Expected result: In your inbox one failure notice will come

Test case for withdraw module in banking project and sending of the emails.

Test case for withdraw module in banking project

Step1: when the balance in the account is nil, try to withdraw some amount (amount>0) should display msg as " insufficient funds in acc"
Step 2: when the account has some balance amount, try to withdraw amount(amount>balance amount in account), should display "insufficient funds in acc"
Step 3: when the account has some balance amount, enter a amount (amount<=balance amount), should withdrawn correct amount from account.
Step 4: when the account has some balance amount, enter the amount as 0, should display msg as withdrawal amount should be > 0 and should be in multiple of hundreds( varies depending on reqs docs).
In the case of Minimum balance mandatory in the Account:
Step 5: When the account has balance amount, try to withdraw whole amount , should display msg as " Minimum balance should be maintained".
Step 6: When the account has balance amount=minimum balance, try to withdraw any amount , should display msg as " Minimum balance should be maintained".

Test cases for sending of the emails?

For testing sending an email you can write test cases for.
1) Performance: By using connections from different ISP's i.e. the speed.
2) If your email id is POP compliant, then check if you can sent it using email clients.
3) If you email can be sent using an attachment.
4) Maximum attachment limit.
5) Maximum mail size.
6) Sending to valid / invalid id's if mail is received / bounced back respectively.

Test Case Samples

Test case for Login window where user name is editable to only upto 8 alpha characters?

1 Enter User Name and press LOGIN Button. User Name= COES. Should Display Warning Message Box "Please Enter User name and Password"

2 Enter Password and press LOGIN Button. Password= COES. Should Display Warning Message Box "Please Enter User name and Password”

3 Enter user Name and Password and press LOGIN Button. "USER = COES AND Password = XYZ" (Wrong user name & password). Should Display Warning Message Box "Please Enter User name and Password"

6 Enter user Name and Password and press LOGIN Button. "USER ="" "" AND Password = "" """(Blank values). Should Display Warning Message Box "Please Enter User name and Password"

7 Enter User Name and Password and press LOGIN Button. "USER = COES AND Password = COES" (Correct user name & password). Should navigate to next page.

8 Enter User Name and Password and press LOGIN Button "USER = ADMIN AND Password = ADMIN" (Correct user name & password). Should navigate to Maintenance page.

Some other test case sample

Test case for withdraw module in banking project
Test cases for sending of the emails?
Testcase for Yahoo Mail Page after login
Test Case For Captcha

Test Case Template

Test Case Template:

What is Test Case?

One or more inputs, execution per-requisites, preconditions, expected result and post conditions developed for a particular objective as per requirements.

Test Case Template may vary from one company to another and one project to another.

A model Test Case Template:

Test Case ID: a unique name (It may consists of Characters and Numbers)

Test Case Name: Name of the Test case

Test Suite ID: Test batch/set/suite ID

Precondition: Required state for Executing the Test Case

Post Condition: State after execution

Steps: Execution steps /Navigation

Expected Result: expected result as per requirements

Actual Results:

Test Results: Pass or Fail

Remarks:

Web Testing

Things to check while doing Web Application Testing

Functionality Testing:

Test for - all the links in web pages, database connection, forms used in the web pages for submitting or getting information from user, Cookie testing.

Check all the links:

Test the outgoing links from all the pages from specific domain under test.
Test all internal links.
Test links jumping on the same pages.
Test links used to send the email to admin or other users from web pages.
Test to check if there are any orphan pages.
Lastly in link checking, check for broken links in all above-mentioned links.

Test forms in all pages:

Forms are the integral part of any web site. Forms are used to get information from users and to keep interaction with them. So what should be checked on these forms?
First check all the validations on each field.
Check for the default values of fields.
Wrong inputs to the fields in the forms.
Options to create forms if any, form delete, view or modify the forms.

Cookies testing:

Cookies are small files stored on user machine. These are basically used to maintain the session mainly login sessions. Test the application by enabling or disabling the cookies in your browser options. Test if the cookies are encrypted before writing to user machine. If you are testing the session cookies (i.e. cookies expire after the sessions ends) check for login sessions and user stats after session end. Check effect on application security by deleting the cookies.

Validate your HTML/CSS:

If you are optimizing your site for Search engines then HTML/CSS validation is very important. Mainly validate the site for HTML syntax errors. Check if site is crawl able to different search engines.

Database testing:

Data consistency is very important in web application. Check for data integrity and errors while you edit, delete, modify the forms or do any DB related functionality. Check if all the database queries are executing correctly, data is retrieved correctly and also updated correctly. More on database testing could be load on DB, we will address this in web load or performance testing below.

Usability Testing:

Test for navigation:

Navigation means how the user surfs the web pages, different controls like buttons, boxes or how user using the links on the pages to surf different pages.
Usability testing includes:
Web site should be easy to use. Instructions should be provided clearly. Check if the provided instructions are correct means whether they satisfy purpose. Main menu should be provided on each page. It should be consistent.

Content checking:

Content should be logical and easy to understand. Check for spelling errors. Use of dark colors annoys users and should not be used in site theme. You can follow some standards that are used for web page and content building. These are common accepted standards like as I mentioned above about annoying colors, fonts, frames etc.
Content should be meaningful. All the anchor text links should be working properly. Images should be placed properly with proper sizes.
These are some basic standards that should be followed in web development. Your task is to validate all for UI testing

Other user information for user help:

Like search option, sitemap, help files etc. Sitemap should be present with all the links in web sites with proper tree view of navigation. Check for all links on the sitemap.
“Search in the site” option will help users to find content pages they are looking for easily and quickly. These are all optional items and if present should be validated.

Interface Testing:

The main interfaces are:
Web server and application server interface
Application server and Database server interface.
Check if all the interactions between these servers are executed properly. Errors are handled properly. If database or web server returns any error message for any query by application server then application server should catch and display these error messages appropriately to users. Check what happens if user interrupts any transaction in-between? Check what happens if connection to web server is reset in between?

Compatibility Testing:

Compatibility of your web site is very important testing aspect. See which compatibility test to be executed:
Browser compatibility
Operating system compatibility
Mobile browsing
Printing options


Browser compatibility:

In my web-testing career I have experienced this as most influencing part on web site testing.
Some applications are very dependent on browsers. Different browsers have different configurations and settings that your web page should be compatible with. Your web site coding should be cross browser platform compatible. If you are using java scripts or AJAX calls for UI functionality, performing security checks or validations then give more stress on browser compatibility testing of your web application.
Test web application on different browsers like Internet explorer, Firefox, Netscape navigator, AOL, Safari, Opera browsers with different versions.

OS compatibility:

Some functionality in your web application is may not be compatible with all operating systems. All new technologies used in web development like graphics designs, interface calls like different API’s may not be available in all Operating Systems.
Test your web application on different operating systems like Windows, Unix, MAC, Linux, Solaris with different OS flavors.

Mobile browsing:

This is new technology age. So in future Mobile browsing will rock. Test your web pages on mobile browsers. Compatibility issues may be there on mobile.

Printing options:

If you are giving page-printing options then make sure fonts, page alignment, page graphics getting printed properly. Pages should be fit to paper size or as per the size mentioned in printing option.

Performance testing:

Web application should sustain to heavy load. Web performance testing should include:
Web Load Testing
Web Stress Testing
Test application performance on different internet connection speed.
In web load testing test if many users are accessing or requesting the same page. Can system sustain in peak load times? Site should handle many simultaneous user requests, large input data from users, Simultaneous connection to DB, heavy load on specific pages etc.

Stress testing:

Generally stress means stretching the system beyond its specification limits. Web stress testing is performed to break the site by giving stress and checked how system reacts to stress and how system recovers from crashes. Stress is generally given on input fields, login and sign up areas.
In web performance testing web site functionality on different operating systems, different hardware platforms is checked for software, hardware memory leakage errors,

Security Testing:

Following are some tests for web security testing:
Test by pasting internal URL directly into browser address bar without login. Internal pages should not open.
If you are logged in using username and password and browsing internal pages then try changing URL options directly. I.e. If you are checking some publisher site statistics with publisher site ID= 123. Try directly changing the URL site ID parameter to different site ID which is not related to log in user. Access should deny for this user to view others stats.
Try some invalid inputs in input fields like login username, password and input text boxes. Check the system reaction on all invalid inputs.
Web directories or files should not be accessible directly unless given download option.
Test the CAPTCHA for automates scripts logins.
Test if SSL is used for security measures. If used proper message should get displayed when user switch from non-secure http:// pages to secure https:// pages and vice versa.
All transactions, error messages, security breach attempts should get logged in log files somewhere on web server.

Web Applications Testing

Now a days Web Applications are getting more popular in the IT Industry, having so many advantages like supporting more clients, no client side installation and accessing from any ware etc…

Types of Web Application
We can categorize web applications in two ways

a) Business Classification

1) Web Sites (They provide information about Organizations or Industries or persons)
2) Web Portals (They are Business gateways, ex: Online Shopping sites, Job portals etc.)
3) Web Applications (They are Service providers (both Free and Paid), apart from information) Ex: Net Banking Applications, Insurance Applications etc…)

b) Technical Classification
1) Intranet Applications (They are private applications, uses local area network (LAN))
2) Internet Applications (They are Public applications, uses Wide area network (WAN)
3) Extranet Applications (They also Private applications over Internet (WAN))


Browser:
It is a Software Application, which retrieves, and Presents information in text, image and voice like different file formats.

The browser is the viewer of a Web Site and there are so many different browsers and browser options that a well-done Web Site is probably designed to look good on as many browsers as possible.

Popular Browsers:
1) Internet Explorer
2) Mozilla Firefox
3) Google Chrome
4) Safari
5) Opera
Etc…

Wednesday 23 November 2011

Selenium

Selenium is a suite of tools used to automate web application testing across many platforms. It is an open source tool developed in Java Script and browser technologies and hence supports all the major browsers on all the platforms. It was developed by ThoughtWorks Inc.

There are 4 components of Selenium:

1) Selenium Core:

This is the original JavaScript-based testing system. It is now used primarily as a component of Selenium Remote Control, but it can also be used as a pure JavaScript/HTML testing system. To run Selenium Core tests, they normally have to be installed on the server on which you want to test.

2) Selenium IDE:

Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox (2+) extension, and allows you to record, edit, and debug tests.

Features:

* Easy record and playback
* Intelligent field selection using IDs, names, or Xpath, as needed
* Autocomplete for all common Selenium commands
* Walk through tests
* Debug and set breakpoints
* Save tests as HTML, Ruby scripts, or any other format
* Support for Selenium user-extensions.js file
* Option to automatically assert the title of every page

Drawbacks:

* The biggest drawback of Selenium IDE is its limitation in terms of browser support. Though Selenium scripts can be used for most of the browsers and operating systems, scripts written using Selenium IDE can be used for only the Firefox browser if it is not used with Selenium RC or Selenium Core.
* Selenese, the simple scripting language used by IDE, is somewhat primitive as it has no conditionals (no "if" statements), and no loops (no "for" statements).
* Does not help with the launching and closing of the browser

Note: Selenium IDE can be configured to repeat a specific test by using flowControl extensions: goto, While, and so on, by installing the selenium-goto extension (download it from http://wiki.openqa.org/pages/viewpageattachments.action?pageId=379). To install it, open the Options panel in the Selenium IDE browser extension and specify the location of the user-extension.js file.


3) Selenium Remote Control:

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser. The RC server also bundles Selenium Core, and automatically loads it into the browser.

Selenium RC comes in two parts:

a) A server, which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.

b) Client libraries for your favorite computer language (Java, .NET, Perl, Python, C#, PHP and Ruby).

Click on selenium for a simplified architectural representation.

As a test suite starts, the following happens

* The client/driver reaches out to the Selenium-Server.
* The Selenium-Server launches a browser (or reuses an old one) with a URL that will load the Selenium core web page.
* The Selenium-Core gets the first instruction from the client/driver (via the HTTP Proxy built into the Selenium RC Server).
* The Selenium-Core acts on that first instruction, typically opening a page of the AUT.
* The web server is asked for that page, and it renders it in the frame/window reserved for it.

4) Selenium Grid:

Selenium Grid runs tests on many servers at the same time, cutting down on the time it takes to test multiple browsers or operating systems.

Installation:

1. Selenium IDE:
* Selenium IDE can be downloaded from http://seleniumhq.org/download/.
* Open Mozilla Firefox.
* Drag and drop the file into it.
* Install it and restart Firefox
* If installed properly, Selenium can be accessed from Tool –> Selenium IDE in your browser toolbar.
2. Selenium RC:
* Selenium RC can be downloaded from http://seleniumhq.org/download/.
* Unzip the file and extract it in the directory of your choice.
* JRE 1.5 or higher version should be installed on your machine. To check the version, type the following command at the dos prompt: java -version.

Thursday 17 November 2011

What is "Automated Testing"?


What is meant by "Automated Testing", is automating the manual testing process currently in use. This requires that a formalized "manual testing process" currently exists in your company or organization. Minimally, such a process includes:

• Detailed test cases, including predictable "expected results", which have been developed from Business Functional Specifications and Design documentation

• A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application
If your current testing process does not include the above points, you are never going to be able to make any effective use of an automated test tool.

So if your "testing methodology" just involves turning the software release over to a "testing group" comprised of "users" or "subject matter experts" who bang on their keyboards in some ad hoc fashion or another, then you should not concern yourself with testing automation. There is no real point in trying to automate something that does not exist. You must first establish an effective testing process.
The real use and purpose of automated test tools is to automate regression testing. This means that you must have or must develop a database of detailed test cases that are repeatable, and this suite of tests is run every time there is a change to theapplication to ensure that the change does not produce unintended consequences.
An "automated test script" is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development. Making effective use of any automated test tool requires at least one trained, technical person – in other words, a programmer.

Automated Testing

The principle of automated testing is that there is a program (which could be a job stream) that runs the program being tested, feeding it the proper input, and checking the output against the output that was expected. Once the test suite is written, no human intervention is needed, either to run the program or to look to see if it worked; the test suite does all that, and somehow indicates (say, by a :TELL message and a results file) whether the program's output was as expected. We, for instance, have over two hundred test suites, all of which can be run overnight by executing one job stream submission command; after they run, another command can show which test suites succeeded and which failed.

These test suites can help in many ways:

1] As discussed above, the test suites should always be run before a
new version is released, no matter how trivial the modifications
to the program.

2] If the software is internally different for different
environments (e.g. MPE/V vs. MPE/XL), but should have the same
external behavior, the test suites should be run on both
environments.

3] As you're making serious changes to the software, you might want
to run the test suites even before the release, since they can
tell you what still needs to be fixed.

4] If you have the discipline to -- believe it or not -- write the
test suite before you've written your program, you can even use
the test suite to do the initial testing of your code. After all,
you'd have to initially test the code anyway; you might as well
use your test suites to do that initial testing as well as all
subsequent tests.

Note also that the test suites not only run the program, but set up
the proper environment for the program; this might mean filling up a
test database, building necessary files, etc.

Why Automation Testing is Important


Everyone  knows  how  important  testing is, and,  with luck, everyone actually does test the software that they release. But do they really?Can  they?  Even  a  simple program often  has many different possiblebehaviors,  some of which only take  place in rather unusual (and hard
to  duplicate)  circumstances.  Even  if  every possible  behavior was tested  when  the program was first released  to the users, what about the  second release, or even a "minor" modification? The feature being modified  will probably be re-tested,  but what about other, seemingly
unrelated,  features  that  may have been  inadvertently broken by the modification?  Will  every unusual test case  from the first release's testing  be  remembered,  much  less  retried,  for  the  new release, especially  if  retrying  the test would require  a lot of preliminary work (e.g. adding appropriate test records to the database)?

This  problem  arose for us several years  ago, when we found that our software  was  getting  so complicated that  testing everything before release  was  a  real  chore, and a good many  bugs (some of them very obvious)  were getting out into the field. What's more, I found that I
was  actually  afraid  to add new features,  concerned that they might break  the rest of the software. It  was this last problem that really drove  home to me the importance of  making it possible to quickly and easily test all the features of all our products.

Wednesday 16 November 2011

What you should known to do good or best performance testing


Well, I have been spending practically all my spare time in designing (and re-designing) a plan for our house that we will build shortly. However, I thought that it would be good to write a post about the knowledge that one needs for successful performance testing.
Performance testing deals with design and execution of tests that examine the performance of the application under test. The application's performance is measured using a number of metrics such as application's response times, application throughput and concurrent users. Many software testers struggle when they begin performance testing/ load testing. This is because performance testing requires familiarity with a number of special concepts as well as proficiency in certain special skills. However, the good news is that you can learn the required concepts, develop the required skills and deliver results in your performance tests successfully. The purpose of this post is not to define the key terms used in performance testing but to introduce them to you. You can search these terms on the web and build your knowledge.
1. Performance testing tools (commercial, open source and custom)
There are numerous performance testing tools available publicly. Some tools are commercial and the others are open source.The full-featured tools provide the functionality to create test scripts, add test data, set up tests, execute the tests and display the results. If the performance testing tool has not been chosen yet, you should evaluate the tools according to your project requirements as per the evaluation process described here.
If you have the time and technical skills, you may even create your own simple tool to help in performance testing.
2. Profilers

Profilers are tools that measure data related to the application's calls or the resources (e.g. memory) used by the application when the application is running.
3. Virtual Users

If your application supports multiple users concurrently (at the same time), you should test your application's performance using multiple users. The users modeled by the tool are called the Virtual Users.
4. Key business transactions
Your application may allow the user a large number of work flows or business transactions. All the work flows/ business transactions may not be important
in performance testing. Therefore, it is common to test using only the important or key business transactions. Refer or solicit your application's performance requirements for guidance in this regard.
5. Workload
Workload is the load on the multi-user applications in terms of virtual users performing different business transactions. For example, in the case of a social networking application, 50 virtual users may be searching contacts, 40 of them may be messaging and 10 may be editing their profiles, all within a period of 30 minutes.
6. Isolation of test environment
In order to get results with confidence, it is critical that your test environment is used only for the purpose of the performance test. This means that no other systems or users should be loading the test environment at the time of the performance test. Otherwise, you may have trouble replicating (or even
understanding) your test results.
7. Modeling (script and test)
You should script the key business transactions as realistically as possible (model) using the performance test tool. You should also design the tests with realistic test settings (e.g. virtual users ramp-up, user network bandwidth, user browser and so on) and the model workload.
8. Test data
It is common for the test scripts to be executed numerous times during one performance test. Therefore, you may need a large amount of test data that does not exhaust during the test. Once a performance test finishes, the application may be full of dummy test data entered by the test. You may need a means of cleaning up this data (for example, by re-installing the build).
9. Server configurations
You should be aware of and control the server configuration (CPU, memory, hard disk, network bandwidth and so on). This is because the application performance depends on the server resources.
10. Network configurations
You should know about the protocols used by your application. You should also know about load balancing (in case, multiple servers are used by your application).
11. Client configurations
You should know the common client configurations (in terms of CPU, memory, network bandwidth, operating system, browser and so on) used by your users. This would help you model realistic tests.
12. Load generators
Depending on the load that you need to generate during the test, you may need one or more load generator machines. One tool may need fewer resources (CPU usage, memory usage etc.) per virtual user and another tool may need more resources per virtual user. You should have sufficient load generation capacity without maxing out your load generator(s) in any way.
13. Performance counters
During the test, you should monitor the chosen performance counters (e.g. % Processor time, Average disk queue length) on your load generators as well as the application's servers. You should choose the performance counters so that you may come to know about the depletion (or near depletion) of any important resource.
14. Response time
You should be aware that the application response time has two components. The server takes time to create a response. Further, the response sent by the server takes time to render on the client.
15. Monitoring
During a performance test, you should be monitoring the test progress, any errors thrown as well as your chosen performance counters on your servers and load generators.
16. Results Analysis
After the completion of a performance test, you should spend time on analyzing the test results. You should check if the test created the required virtual users, generated the required load and ran to completion. You should check the errors thrown during the test. If you see any unusual results, you should form a conjecture to explain those results and look at the data carefully to either accept or reject your assumption.
It takes practice to be adept at analyzing performance tests well.
17. Reporting
You should be comfortable with reporting performance test results. It is common for the performance test reports to contain present/ past metrics as well as charts.
If you make some effort, it is not difficult to educate yourself with the knowledge required for performance testing. Let me know if you wish to discuss any specific topic in performance testing.

Stress Testing

Stress testing is a special type of performance testing. It is used to stress (or in other words, put a high load on) the system to determine its highest operational capacity. Stress tests establish the highest load that the system is able to withstand while still being operational (and not throwing too many errors). The additional benefit of stress testing is that it exposes the resources that run out the fastest when the load on the system approaches its maximum sustainable value. It is common to design a series of load tests with increasing loads to find out the maximum sustainable load.

Here are the tips to perform stress testing correctly but quickly:

1. Before you begin work on stress testing your system, you should be aware of the goals of your stress test. For example, your goal could be to determine the maximum throughput of your application for a given business transaction/ workflow on the given infrastructure or your goal could be to determine the maximum number of concurrent users with a given workload and the given infrastructure.

2. It is crucial that you identify the correct performance testing tool for your stress test. Otherwise, you would be forced to make compromises in your tests. See How to evaluate automated software test tools quickly?
Also see Open Source Software Test Tools.

3. You would need to model test script(s) according to the business transactions specified or implied in your goals.

4. If your scripts are correct, a large number of business transactions would execute during your tests. Therefore, ensure that you have a large set of test data that does not exhaust during the test. You may consider sourcing any existing test data from your application's database. Another way of quickly generating some test data is to use Microsoft Excel to extend a series. There are also free test data generators available on the web.

5. As with other tests, you should have an isolated (as far as possible) environment for conducting your tests. Unless you stress test a system on a stand-alone machine, your test environment may consist of the client machine(s), the load generator machine(s), the server(s) (web server(s), application server(s), database server(s) and so on) and the connectivity between the above. You would need your performance testing tool installed on at least one machine. You would set up your test, run it and analyze the test results from this machine. Depending on your choice of the performance testing tool, other load generator machine(s) may or may not require the full performance testing tool installed on them.

6. When you set up your first load test, you should model it realistically. For example, you should add the required scripts to the test, set up the specifications of the Virtual Users (e.g. initial VU, ramp-up, constant load, ramp-down and distribution of VU assigned to each script etc.) and give the run-time settings (e.g. test duration, time lag between iterations, network bandwidth distribution of the VU, their browser distribution etc.). You would really be limited by the features provided by the chosen performance testing tool.
You should specify a low load in your first load test. This is when you could check if your script(s) work correctly (e.g. the script reads the test data, it accesses the system and interacts with it, it is able to send data to the system which is accepted by the system and is able to send/ receive data to/ from other script(s) as required). If you have multiple scripts in your test, you could check if the scripts run in the correct sequence or in parallel, as specified. When your test is running, you should check the performance of each machine in your
test environment e.g. check that the server(s) and load generator(s) have become busier but not to their limit. On Microsoft Windows machines, you may use the PerfMon (Start > Run > PerfMon) tool to add your desired counters and monitor them during the test. See other tips in the post, Performance testing using HP LoadRunner software.

7. After establishing that your load test is valid, you should increase the load systematically. Do not go "big bang" and deploy the load that you think is the maximum sustainable load. For example, if you think that your system may support up to 1000 concurrent Virtual Users, start with a load test with 100 VU and keep on adding 100 VU every 5 minutes. You may find that your system supports 800 concurrent VU but crashes with a load of 900 concurrent VU. After resetting your test environment, you should then start with a load test with 800 VU and keep on adding 10 VU every 30 seconds. Then you may find that your system supports 850 VU but crashes with a load of 860 concurrent VU. Repeat the last test at least another couple of times to confirm your test result.

8. If you recall, I mentioned that the additional benefit of a stress test is to identify bottlenecks or resources that run out the fastest when the load is increased. You should analyze the test results of several of your last tests to find out about these bottlenecks. Maybe it is the network which is clogged, or your web server runs out of available memory. Confirm your finding by repeating the last few tests. Your system's capacity may be improved by increasing the resource that runs out first. Ensure that you include the details of the important bottlenecks observed during your stress test in your test report.

Performance Testing using HP LoadRunner software

There is an excellent white paper titled, "HP LoadRunner software—tips and tricks for configuration, scripting and execution". You can read this white paper that contains numerous tips as well as code samples. There are tips to create a high level performance test plan and tips for scripting, building scenarios, data handling and test execution. You can see this white paper here.

Now let us turn our attention to the analysis of the performance test results. As I have mentioned in my earlier post, What knowledge do you need to have in order to do good or best performance testing?, after you run your test, you should first check if the performance test itself was valid. Initially, you may not be able to realize whether the errors you see are errors thrown due to the limitations of your application/ infrastructure or due to defects in your test environment. Therefore, you should start out with a simple and small test (e.g. by using a small number of virtual users, few transactions per second, short test duration and so on). Examine any errors raised by your test. Try to establish the reason of the most common errors and change your script(s), test or tool/ infrastructure configuration accordingly. However, you should make only one change at a time to your script(s) or test. The benefit of this is that if your change were incorrect, you would be able to rollback your change and try something else.

When you are able to run a simple and small test successfully (with minimal errors), it is time to develop your test. Again, do this incrementally. Run the test once, analyze the results and address any problems with the test environment before you make another change to it. Continue to develop your test until you have your full test available with you.

A test running with no errors or minimal errors may still be incorrect. Look for the supporting data, for example:
1. Does the test show that it has generated the Virtual Users specified by you?
2. Do metrics like transactions per second climb up in line with the ramp-up specified by you?
3. If the test puts a substantial load on your application, does your application slow down during the test?
4. Do your load generators become busier during the test?
5. If the test is supposed to create test data in the application, do you see this data in the application after the test is complete?

Even with a good handle on the errors and supporting data from multiple sources, you should not run your test just once. Run the same test multiple times and see if the results match with each other (more or less). If the results do not match, you should find out the reason for the mismatch and address it in your test environment.

Finally, if your test runs multiple times as you expect, it is time to check your test results against your application's performance requirements.

As you can now see, the performance test process (test preparation, test development, test execution and test analysis) is iterative.

Tips to do Database Testing

Many (but not all) applications under test use one or more databases. The purposes of using a database include long-term storage of data in an accessible and organized form. Many people have only a vague idea about database testing. If you are serious about learning database testing, then read on...

Firstly, we need to understand what is database testing? As you would know, a database has two main parts - the data structures (the schema) that store the data AND the data itself. Let us discuss them one by one.

The data is stored in the database in tables. However, tables may not be the only objects in the database. A database may have other objects like views, stored procedures and functions. These other objects help the users access the data in required forms. The data itself is stored in the tables. Database testing involves finding out the answers to the following questions:

Questions related to database structure
1. Is the data organized well logically?
2. Does the database perform well?
3. Do the database objects like views, triggers, stored procedures, functions and jobs work correctly?
4. Does the database implement constraints to allow only correct data to be stored in it?
5. Is the data secure from unauthorized access?


Questions related to data
1. Is the data complete?
2. Is all data factually correct i.e. in sync with its source, for example the data entered by a user via the application UI?
3. Is there any unnecessary data present?


Now that we understand database testing, it is important to know about the 5 common challenges seen before or during database testing:

1. Large scope of testing
It is important to identify the test items in database testing. Otherwise, you may not have a clear understanding of what you would test and what you would not test. You could run out of time much before finishing the database test.
Once you have the list of test items, you should estimate the effort required to design the tests and execute the tests for each test item. Depending on their design and data size, some database tests may take a long time to execute. Look at the test estimates in light of the available time. If you do not have enough time, you should select only the important test items for your database test.

2. Incorrect/ scaled-down test databases
You may be given a copy of the development database to test. This database may only have little data (the data required to run the application and some sample data to show in the application UI). Testing the development or test or staging databases may not be sufficient. You should also be testing a copy of the production database.

3. Changes in database schema and data
This is a particularly nasty challenge. You may find that after you design a test (or even after you execute a test), the database structure (the schema) has been changed. This means that you should be aware of the changes made to the database during testing. Once the database structure changes, you should analyze the impact of the changes and modify any impacted tests.
Further, if your test database is being used by other users, you would not be sure about your test results. Therefore, you should ensure that the test database is used for testing purpose only.
You may also see this problem if you run multiple tests at the same time. You should run one test at a time at least for the performance tests. You do not want your database performing multiple tasks and under-reporting performance.

4. Messy testing
Database testing may get complex. You do not want to be executing tests partially or repeating tests unnecessarily. You should create a test plan and proceed accordingly while carefully noting your progress.

5. Lack of skills
The lack of the required skills may really slow things down. In order to perform database testing effectively, you should be comfortable with SQL queries and the required database management tools.

Next, let us discuss the approach for database testing. You should keep the scope of your test as well as the challenges in mind while designing your particular test design and test execution approach. Note the following 10 tips:

1. List all database-specific requirements. You should gather the requirements from all sources, particularly technical requirements. It is quite possible that some requirements are at a high level. Break-down those requirements into the small testable requirements.

2. Create test scenarios for each requirement as suggested below.

3. In order to check the logical database design, ensure that each entity in the application e.g. actors, system configuration are represented in the database. An application entity may be represented in one or tables in the database. The database should contain only those tables that are required to represent the application entities and no more.

4. In order to check the database performance, you may focus on its throughput and response times. For example, if the database is supposed to insert 1000 customer records per minute, you may design a query that inserts 1000 customer records and print/ store the time taken to do so. If the database is supposed to execute a stored procedure in under 5 seconds, you may design a query to execute the stored procedure with sample test data multiple times and note each time.

5. If you wish to test the database objects e.g. stored procedures, you should remember that a stored procedure may be thought of as a simple program that (optionally) accepts certain input(s) and produces some output. You should design test data to exercise the stored procedure in interesting ways and predict the output of the stored procedure for every test data set.

6. In order to check database constraints, you should design invalid test data sets and then try to insert/ update them in the database. An example of an invalid data set is an order for a customer that does not exist. Another example is a customer test data set with an invalid ZIP code.

7. In order to check the database security, you should design tests that mimic unauthorized access. For example, log in to the database as a user with restricted access and check if you can view/ modify/ delete restricted database objects or view or view and update restricted data. It is important to backup your database before executing any database security tests. Otherwise, you may render your database unusable.
You should also check to see that any confidential data in the database e.g. credit card numbers is either encrypted or obfuscated (masked).

8. In order to test data integrity, you should design valid test data sets for each application entity. Insert/ update a valid test data set (for example, a customer) and check that the data has been stored in the correct table(s) in correct columns. Each data in the test data set should have been inserted/ updated in the database. Further, the test data set should be inserted only once and there should not be any other change in the other data.

9. Since your test design would require creating SQL queries, try to keep your queries as simple as possible to prevent defects in them. It is a good idea for someone other than the author to review the queries. You should also dynamically test each query. One way to test your query is to modify it so that it just shows the resultset and does not perform the actual operation e.g. insert, delete. Another way to test your query is to run it for a couple of iteration s and verify the results.

10. If you are going to have a large number of tests, you should pay special attention to organizing them. You should also consider at least partial automation of frequently run tests.

Now you should know what database testing is all about, the problems that you are likely to face while doing database testing and how to design a good database test approach for the scope decided by you. Why don't you consider professional database testing for your application? Let me know your experiences.

What are the responsibilities of a Software Test Lead

Common responsibilities of a Software Test Lead include the ones given below.
1. Be updated on the latest testing techniques, strategies, testing tools/ test frameworks and so on
2. Be aware of the current and upcoming projects in the organization
3. Review and analyze the project requirements
4. Plan and organize the knowledge transfer to the Software Test Engineers and self
5. Collect the queries related to the requirements and get them resolved by the business person (e.g. the client, business analyst, product manager or project manager) assigned to the project
6. Plan, organize and lead the testing kick-off meeting
7. Scope the required tests
8. Design the required test strategy in line with the scope and organization standards
9. Create the software test plan, get it reviewed and approved/ signed-off by the relevant stakeholders
10. Evaluate and identify the required test automation and test management tools
11. Estimate the test effort and team (size, skills, attitude and schedule)
12. Create the test schedule (tasks, dependencies and assigned team members)
13. Identify the training requirements of the Software Test Engineers
14. Identify any test metrics to be gathered
15. Communicate with the client or on site/ offshore team members, as required
16. Review the test cases and test data generated by the Software Test Engineers and get them to address the review comments
17. Track the new/ updated requirements in the project and modify testing artifacts accordingly
18. Determine, procure, control, maintain and optimize the test environment (hardware, software and network)
19. Get information on the latest releases/ builds from the development team/ the client
20. Create and maintain the required test automation framework(s)
21. Administer the project in the test management system
22. Administer the Application under test (e.g. add users for the tests), as required
23. Assign tasks to the Software Test Engineers based on the software test plan
24. Check the status of each assigned task daily and resolve any issues faced by the team members with their tasks
25. Ensure that each team member is optimally occupied with work (i.e. each Software Test Engineer should not be too overloaded or too idle)
26. Re-assign the testing tasks, as required
27. Track the assigned tasks with respect to the software test plan and the project schedule
28. Review the test automation created by the Software Test Engineers and get them to address the review comments
29. Own and maintain the test automation suite of the project
30. Schedule and execute the test automation on the project
31. Review defect reports and assign valid defects to the relevant developer/ development manager
32. Assign returned defect reports and assist the concerned Software Test Engineer, as required
33. Ensure the resolved defects are re-tested
34. Consolidate and report test results to the concerned stakeholders
35. Be approachable and available to the Software Test Engineers, as required by them
36. Update the software test plan, as required
37. Ensure that the test cases are updated by the Software Test Engineers, as required
38. Ensure that the test automation is updated based on the updated test cases
39. Gather the decided test metrics
40. Escalate and obtain resolution of the issues related to the test environment and team
41. Plan, organize and lead team meetings and ensure action is taken based on the team discussions
42. Plan and organize training for the Software Test Engineers
43. Review the status reports of the Software Test Engineers
44. Review the time logged by the Software Test Engineers for various activities
45. Report the status to the stakeholders (e.g. the client, project manager/ test manager and the management)
46. Keep the Software Test Engineers motivated
47. Improve the test process based on the suggestions by others and own judgment
48. Manage own energy level and time

Friday 11 November 2011

A/B Testing of web applications

Many websites use a type of software testing called A/B testing or split testing. The objective of this testing is to determine and positively influence the user experience on the website. It involves testing distinct design layout options of the website. A/B testing is also performed on non-web elements of the website such as emails. Many large companies use A/B testing. So, what is A/B testing?

A/B testing determines the better of two content or design options on real users using web analytics tools. It requires an existing benchmark to measure against. For example, let us say that you test an ecommerce website. Looking at the website logs or analytics, you know that only 20% of the users who start a checkout process actually complete it. You suspect that your multi-page checkout process could be a cause of checkout abandonment. Instead of directly changing the checkout process to single-page, you decide to execute an A/B test. For this, you set up two checkout options - the current multi-page and the new single-page checkout. From now on, 50% of the users who start the checkout get the multi-page checkout option and the others get the single-page checkout. In order to not confuse the users, you record which option was presented to which user and continue to provide the same option to the users on their repeat visits too. You monitor the test results for a month. Then you analyze the results and take necessary action.

Possible test results
1: Multi-page Complete = 10%, Single-page Complete=30%: This is strongly in favor of the single-page option. So, you decide on deploying this option.
2. Multi-page Complete = 18%, Single-page Complete=22%: This is a minor difference. Maybe, the cause of low checkout completion is not the number of pages. So, you decide to look for other elements and then run a different A/B test.
3. Multi-page Complete = 28%, Single-page Complete=12%: This is in favor of the existing option. So, this is not the cause of low check out completion. You decide to test another element e.g. the actual text on the checkout pages.
4. Multi-page Complete = 10%, Single-page Complete=12%: This is marginally in favor of the new option. But, more urgently you need to find out the cause of both options performing below the benchmark.

Points to keep in mind
a. The two options should have only a very limited number of differences. If there are many differences, it is hard to pin-point the improved element with accuracy.
b. The sample sizes in A/B testing should be statistically significant. For example, the results based on an A/B test on 200 users is realistic. A test on just 5 users is not.
c. The two options should be tested simultaneously, rather than one after the other. If tested consecutively, it is possible for other factors (e.g. changed user patterns/ demographics) come into play and this may skew the results.
d. Instead of just two options, multiple options can be tested. In this case, it will be called A/B/N testing (for N options).
e. Each and every element of the web application interface is a candidate for A/B testing. The elements to be considered include home page, actual text, font faces and sizes, colors, images, links, linked pages, placement of elements and so on.
f. A/B testing should be done on an ongoing basis. After improving one element of the user experience, the next should be targeted and tested.

Why your Bugs are rejected

Scenario: You test your application. You discover some bugs. You are happy. You craft your bug reports and log them in the bug tracking system. The next day, you find that a number of the bugs submitted by you have been rejected by the developers.

What happened?

You may face bug rejections even when you are "seemingly" careful in your testing and bug reporting activities. Of course, you want your tests to be taken seriously. Do you know the various reasons why your bugs may be rejected? Reflect on the reasons given below and consider them when reporting bugs.

Bug rejection reasons related to requirements
1. There is no documented requirement that mentions the result expected in your bug report.
2. You are not aware of all the (minor) details (and caveats) of a requirement.
3. The requirement that you thought was not implemented is indeed implemented in another way. But you are not aware of this decision.
4. You are not aware about the changes to a particular requirement.
5. Your bug report relates to some work that has not started yet. Or, the work has started but has not been submitted for test yet.
6. Your bug report refers to some dummy test data in the application. This bug would not exist when the real/ realistic data is set up in the application.
7. The bug is in an area of the application not within the scope of development e.g. external systems, external content etc.
8. It has been like this since the beginning or for a long time OR the users have always accepted this result.

Bug rejection reasons related to test environment
9. The settings (e.g. language, region and fonts) on your test system are incorrect.
10. The application settings (e.g. settings in the administration section) used by you are incorrect.
11. Some required files are missing on your test system.
12. The required programs are missing on your test system. Or, you have the incorrect versions of the required programs.
13. The application does not support the platform (e.g. operating system and its versions or browser and its versions) tested by you.
14. You have tested against an old build. The bug is not present in the current build submitted for test.

Bug rejection reasons related to test data
15. The test data used by you is unrealistic.

Bug rejection reasons related to the test steps/ test case
16. The steps described in your bug report are unrealistic.

Bug rejection reasons related to the bug report itself
17. The Title (Summary) and Description of the bug report are mismatched.
18. It is not possible to understand your bug report (due to incomplete information provided or other reasons).
19. It is a duplicate bug report. For example, it is a complete duplicate of another bug report OR it describes the same problem that occurs in another area of the application as given in a valid bug report OR it describes a problem that is a part of the problem described in another valid bug report)

Bug rejection reasons related to management decisions
20. A long time after you submitted the bug, it is decided that it is not feasible to put in the effort required to fix it.
21. Due to strategic changes in the plan, all bug reports related to specific application functionality are rejected. Your bug report is one of those that are rejected.

Other reasons
22. The bug report describes a problem that is due to another bug. When this other bug would be fixed, the problem mentioned in your bug report would not exist.
23. The bug is too minor.

Once your bug is rejected, you would do well to remain calm. Read your bug report carefully and try to reproduce the bug. Then, analyze the reason for the bug rejection, establish whether it is a valid bug or not and take appropriate action. If it really an invalid bug, treat this event as a learning experience. If you think that it is a valid bug, you should confidently take further action in support of your bug report. Either way, you would become more mature as a tester. However, remain motivated and enthusiastic even when you mature. Motivation and enthusiasm play a big part in your testing success.
Have you ever had a bug rejected? What was the reason? Share it in your comment.

How To Find More or Better Bugs

Well, we know that we do not find all the bugs in the application under test (given that the application at hand is not simple). However, we do want to discover and report the most and the best bugs that we can. You need more ideas if you want to find more or better bugs than you do at present. Here are some tips.

Tip 1. Review the application's requirements often. You may notice that no or partial test cases exist for certain requirements. You may find bugs when you test for these requirements. Keep abreast with the change requests/ changes to requirements. Be quick and you may find bugs immediately after a requirement change has been first implemented.

Tip 2. It is possible that you have positive test cases only. If so, create negative test cases for each requirement and execute them.

Tip 3. Execute each test case with a variety of interesting test data (generated for example by the boundary-value analysis technique or pair-wise testing technique).

Tip 4. Test the interface of your application with external systems carefully. This is where a number of bugs may exist.

Tip 5. Another place to look for bugs is the application settings (since this is one area that may be rarely tested in fear that the application may stop working correctly or stop working altogether).

Tip 6. Repeat your tests using other supported client configurations (other CPU/RAM/operating systems/screen resolutions/browsers etc.)

Tip 7. Look at the previous bug reports against the same application or similar applications. See if you can test your application using the ideas or information contained in the previous bug reports.

Tip 8. Do not ignore the cosmetic bugs. If they would inconvenience a user, they should be reported and fixed.

Tip 9. Create a list of items that you would like to test if you had the time. When you test your application, you may find yourself taking mental notes of things that you would like to test later. Jot these things in your list. Come back to the application another time and test the things in your list against all areas of the application.

Tip 10. In case you are working as part of a testing/ QA team, do not restrict yourself to the areas that you are supposed to test. You may find bugs in the other areas. This would also have the nice side-effect of increasing your overall knowledge of the application.

Tip 11. Instead of rushing through your tests at top speed, slow down (I know it is difficult). You would give yourself time to think more clearly. You would start to observe things about your application that you did not observe before.

Tip 12. Finally, take pride in the bugs reported by you. Feel free to mention an interesting bug that you found to a team member.

A word of warning. Do not get carried away if you suddenly start finding a lot more/ better bugs. You should still take the time to contemplate each bug, reproduce it, match it to the requirement and carefully create a good bug report.  

Thursday 10 November 2011

How to do database testing

Many (but not all) applications under test use one or more databases. The purposes of using a database include long-term storage of data in anaccessible and organized form. Many people have only a vague idea about database testing. If you are serious about learning database testing, then read on...

Firstly, we need to understand what is database testing? As you would know, a database has two main parts - the data structures (the schema) that store the data AND the data itself. Let us discuss them one by one.

The data is stored in the database in tables. However, tables may not be the only objects in the database. A database may have other objects like views, stored procedures and functions. These other objects help the users access the data in required forms. The data itself is stored in the tables. Database testing involves finding out the answers to the following questions:

Questions related to database structure
1. Is the data organized well logically?
2. Does the database perform well?
3. Do the database objects like views, triggers, stored procedures, functions and jobs work correctly?
4. Does the database implement constraints to allow only correct data to be stored in it?
5. Is the data secure from unauthorized access?


Questions related to data
1. Is the data complete?
2. Is all data factually correct i.e. in sync with its source, for example the data entered by a user via the application UI?
3. Is there any unnecessary data present?


Now that we understand database testing, it is important to know about the 5 common challenges seen before or during database testing:

1. Large scope of testing
It is important to identify the test items in database testing. Otherwise, you may not have a clear understanding of what you would test and what you would not test. You could run out of time much before finishing the database test.
Once you have the list of test items, you should estimate the effort required to design the tests and execute the tests for each test item. Depending on their design and data size, some database tests may take a long time to execute. Look at the test estimates in light of the available time. If you do not have enough time, you should select only the important test items for your database test.

2. Incorrect/ scaled-down test databases
You may be given a copy of the development database to test. This database may only have little data (the data required to run the application and some sample data to show in the application UI). Testing the development or test or staging databases may not be sufficient. You should also be testing a copy of the production database.

3. Changes in database schema and data
This is a particularly nasty challenge. You may find that after you design a test (or even after you execute a test), the database structure (the schema) has been changed. This means that you should be aware of the changes made to the database during testing. Once the database structure changes, you should analyze the impact of the changes and modify any impacted tests.
Further, if your test database is being used by other users, you would not be sure about your test results. Therefore, you should ensure that the test database is used for testing purpose only.
You may also see this problem if you run multiple tests at the same time. You should run one test at a time at least for the performance tests. You do not want your database performing multiple tasks and under-reporting performance.

4. Messy testing
Database testing may get complex. You do not want to be executing tests partially or repeating tests unnecessarily. You should create a test plan and proceed accordingly while carefully noting your progress.

5. Lack of skills
The lack of the required skills may really slow things down. In order to perform database testing effectively, you should be comfortable with SQL queries and the required database management tools.

Next, let us discuss the approach for database testing. You should keep the scope of your test as well as the challenges in mind while designing your particular test design and test execution approach. Note the following 10 tips:

1. List all database-specific requirements. You should gather the requirements from all sources, particularly technical requirements. It is quite possible that some requirements are at a high level. Break-down those requirements into the small testable requirements.

2. Create test scenarios for each requirement as suggested below.

3. In order to check the logical database design, ensure that each entity in the application e.g. actors, system configuration are represented in the database. An application entity may be represented in one or tables in the database. The database should contain only those tables that are required to represent the application entities and no more.

4. In order to check the database performance, you may focus on its throughput and response times. For example, if the database is supposed to insert 1000 customer records per minute, you may design a query that inserts 1000 customer records and print/ store the time taken to do so. If the database is supposed to execute a stored procedure in under 5 seconds, you may design a query to execute the stored procedure with sample test data multiple times and note each time.

5. If you wish to test the database objects e.g. stored procedures, you should remember that a stored procedure may be thought of as a simple program that (optionally) accepts certain input(s) and produces some output. You should design test data to exercise the stored procedure in interesting ways and predict the output of the stored procedure for every test data set.

6. In order to check database constraints, you should design invalid test data sets and then try to insert/ update them in the database. An example of an invalid data set is an order for a customer that does not exist. Another example is a customer test data set with an invalid ZIP code.

7. In order to check the database security, you should design tests that mimic unauthorized access. For example, log in to the database as a user with restricted access and check if you can view/ modify/ delete restricted database objects or view or view and update restricted data. It is important to backup your database before executing any database security tests. Otherwise, you may render your database unusable.
You should also check to see that any confidential data in the database e.g. credit card numbers is either encrypted or obfuscated (masked).

8. In order to test data integrity, you should design valid test data sets for each application entity. Insert/ update a valid test data set (for example, a customer) and check that the data has been stored in the correct table(s) in correct columns. Each data in the test data set should have been inserted/ updated in the database. Further, the test data set should be inserted only once and there should not be any other change in the other data.

9. Since your test design would require creating SQL queries, try to keep your queries as simple as possible to prevent defects in them. It is a good idea for someone other than the author to review the queries. You should also dynamically test each query. One way to test your query is to modify it so that it just shows the resultset and does not perform the actual operation e.g. insert, delete. Another way to test your query is to run it for a couple of iteration s and verify the results.

10. If you are going to have a large number of tests, you should pay special attention to organizing them. You should also consider at least partial automation of frequently run tests.

Now you should know what database testing is all about, the problems that you are likely to face while doing database testing and how to design a good database test approach for the scope decided by you. Why don't you consider professional database testing for your application? Let me know your experiences.