Agile Development and Testing ----------------------------- PyCon '06, Feb 23rd, Addison, TX **Grig Gheorghiu, grig@gheorghiu.net** **C. Titus Brown, titus@caltech.edu** http://agiletesting.blogspot.com/ http://www.advogato.org/person/titus/ **All of our code, all of our tests, and links to buildbot and other external resources will be made available on http://agile.idyll.org/ by the end of the conference.** Introduction 1. what is agile? * the "testing pyramid" * functionality implemented as "stories" * automated tests * collaboration/"whole team" approach * "tracer bullet" development * "Test-Enhanced Development" 2. lessons learned: * continuous integration is key * remote pair programming works well * whole team approach to both development and testing * no code "thrown over the wall" to QA MailOnnaStick 1. Overview: browse, search, and comment across multiple sources of e-mail. 2. Technologies: * CherryPy * Durus * Commentary * jwzthreading * py.lib logging, xml generation * quixote.html HTML escaping code 3. Implementation timeline * started with simple Web interface to browse mailboxes * added search function * added commenting function * refactored to allow for multiple mailboxes per mail source (Mail dir, IMAP); * currently refactoring to allow for better database storage * next step: better choice? or use? of database. 4. Basic architecture: mos, mosweb, mailsearch 5. Demo. Using Trac and Subversion for collaborative development 1. Trac * Wiki * defect tracking system * source browser * roadmap * can link between everything * timeline is very useful 2. Subversion * source code control system similar to CVS but more advanced * standard layout: trunk, branch, tags * lots of tools available * some sort of code management system is essential. period. * SvnReporter does a nice job of e-mail notifications, etc. Unit testing: 1. theory * completely automated tests that should run *quickly*; * test small, individual "units" of function; * generally test functionality of a specific interface or function * generally don't test full functional "path" through code * may take some setup (load/reset database; build mock object; etc.) in advance of test. 2. using nose: unit test structure * easy to build ad hoc unit tests, but nose gives you a *structure* within which you can easily select subsets of tests, add tests, and see what tests didn't pass. * based on concepts from py.test; written by Jason Pellerin. * some examples * built on top of unittest.py, so can integrate into setup.py * path handling is a bitch... 3. running subtests and doing timing stuff * very useful command line option to select a subset of tests * hacking nose to do timing analysis/display. 4. lessons learned: * the easier to run the better * unit tests serve as explicit interface documentation * writing unit tests for other people's code is a good way to learn the code! FitNesse Acceptance Testing 1. FitNesse * more user friendly variant of Ward Cunningham's FIT framework * "business facing" or "customer facing" tests, as opposed to "code facing" tests (e.g. unit tests) * tests are expressed as stories ("storytests") * higher level than unit tests * FitNesse tests make sure you "write the right code" * unit tests make sure you "write the code right" * wiki format encourages collaboration * Fit/FitNesse brings together business customers, developers and testers, and it forces them to focus on the core business rules of the application. * James Shore: "Done right, FIT fades into the background" 2. Writing FitNesse tests * PyFIT is Python port of FIT/FitNesse * tests are written in tabular format, with inputs and expected outputs * fixtures are a thin layer of glue code that tie the test tables to the application * tests can be executed from both the wiki and the command line. 3. lessons learned: * acceptance tests written with FitNesse can be seen as an alternative GUI into the business logic of the applications * GUI tests are fragile and need frequent changes to keep up with changes in the UI Regression testing with TextTest 1. TextTest is a tool for "behavior based" acceptance testing * generates and then compares log files to known "gold standard", displays differences. * documentation on project's home page is plentiful, but lacks howtos 2. Lessons learned: * log at multiple levels to separate, discrete log files * we're not really using TextTest right. twill functional tests; coverage analysis; profiling. 1. theory: scripting remote HTTP. 2. using twill to browse other Web sites. * twill scripts are simple to write (examples) * interactive browsing demo (example) * twill tests are just scripts that should not fail * extending twill with Python is easy (example) * other packages: mechanize, webtest, webunit, zope.testbrowser, mechanoid. 3. writing automated twill tests: setup/teardown and WSGI intercept (FIGURE) * setup/teardown for a Web server is annoying. * wsgi_intercept lets you run everything in a single process: coverage and profiling are made *much* easier. 4. twill tests can provide extensive code coverage to your Web code. 5. twill can also automate a particular user path through your site, letting you profile and enhance specific targetted site functions. Testing AJAX Web sites with Selenium 1. functional/acceptance testing at the user interface level for Web/AJAX application * uses an actual browser driven via JavaScript! * needs to be deployed on the same server as the app under test for JS security reasons 2. Writing Selenium tests * tests written as HTML tables * tables are small programs, with actions etc. * tools exist for recording tests: Selenium IDE, XPath Checker and XPather Firefox extensions * also a "driven" mode where an external program can drive Selenium via XML-RPC * Selenium tests can be added to continuous integration process 3. Lessons learned * Selenium tests are fairly brittle, like all GUI tests * Not fun to write (although IDEs help!) * Only known way for mankind to test AJAX! doctest 1. testing with doctest * "literate testing" or "executable documentatioon" * unit tests expressed as stories with context * many projects generate documentation from doctests (Django, Zope3) * epydoc makes it trivial to show! * test list -- set of unit tests for a given module; * test map -- set of unit tests functions that exercise a given application function/method * easy to generate automatically 2. Lessons learned * unit test duplication (unit tests, doctests, etc.) is not necessarily a bad thing buildbot I: continuous integration 1. Continuous integration * automate time consuming tasks, get immediate *public* feedback about results of changes * the more often you build & test your software, the better. 2. Using buildbot * written in Twisted; hard to configure but works really well * can be deployed behind Apache for access control purposes * master/slave on multiple servers * run slaves with as many different hardware/software combos as you can * run as many types of tests as possible * (the more aspects of the app you cover, the better protected you are) 3. Lesson learned * running unit/acceptance tests as separate user/different environment will uncover *all kinds* of environment specific issues. trust us. buildbot II: automating everything, and recording everything. 1. automating Selenium UI tests with VNC and Firefox. 2. collating logs and test results. * provides a useful, detailed *record* of where things succeeded, failed. 3. recording logs and test results across runs: Web, billboard. * refer back to prior runs * billboard: get an integrated view of all of the tests Wrap up & Conclusions 1. Holistic testing is the way to go. No one test type does it all... * unit testing of basic code * Functional/acceptance testing of business/user logic * regression testing 2. Make it as easy as humanly possible to run all these types of tests automatically, **or they will not get run**