[转帖]Proposed Software Evaluation & Test KPA - Bender & Associates

翔羽 · 发表于 2006-4-2 08:49:45

KPA REVIEW GROUP The following have been gracious enough to be reviewers of this proposed Testing KPA.  I want to thank them for their insights and contributions.  However, any problems or omissions the reader may find with this document I take full responsibility for.  This is very much a work in progress.

lease feel free to contact me with suggestions for improving it. Boris Beizer - Independent testing consultantGreg Daich - STSCDave Gelperin - SQEBill Hetzel - SQECapers Jones - SPRJohn Musa - ATTWilliam Perry - QAIRobert Poston - IDE   The original version of the Evaluation and Testing KPA was sponsored by Xerox Corporation.  They have graciously allowed us to distribute it to the software community.  The key contact is: David Egerton800 Phillips RoadBuilding 129Webster, NY 14580(716) 422-8822 <h1 style="MARGIN: 12pt 0cm 3pt; TEXT-INDENT: 0cm;"><a name="_Toc353863396"></a><a name="_Toc341669239"></a><a name="_Toc340226053"></a><a name="_Toc339184244"></a><a name="_Toc338414792">1.     INTRODUCTION</a></h1>  The objective of this document is to present a proposal that Evaluation and Test become a Key Process Area (KPA) in the SEI Capability Maturity Model (CMM). The first section addresses the scope of what is meant by evaluation and test.  The second section identifies the justifications for making this a separate KPA.  The third section presents the proposed KPA definition including: definition, goals, commitment to perform, activities performed, measurements and analysis, and verifying implementation.  The final section addresses integrating this KPA with the existing KPA’s.  This includes identifying which level to assign it to and some repackaging suggestions for existing KPA’s. <h1 style="MARGIN: 12pt 0cm 3pt; TEXT-INDENT: 0cm;"><a name="_Toc353863397"></a><a name="_Toc341669240"></a><a name="_Toc340226054"></a><a name="_Toc339184245"></a><a name="_Toc338414793">1.     DEFINING EVALUATION AND TEST</a></h1>  Evaluation is the activity of verifying the various system specifications and models produced during the software development process.  Testing is the machine based activity of executing and validating tests against the code.  Most software organizations define evaluation and test very narrowly.  They use it to refer to just the activities of executing physical test cases against the code.  In fact, many companies do not even assign testers to a project until coding is well under way.  They further narrow the scope of this activity to just function testing and maybe performance testing. This view is underscored in the description of evaluation and test in the current CMM.  It is part of the Software Product Engineering KPA.  The activities in this KPA, activities 5, 6, and 7, only use code based testing for examples and only explicitly mention function testing.  Other types of testing are euphemistically referenced by the phrase “...ensure the software satisfies the software requirements”.

eople who build skyscrapers, on the other hand, thoroughly integrate evaluation and test into the development process long before the first brick is laid.  Evaluations are done via models to verify such things as stability, water pressure, lighting layouts, power requirements, etc.  The software evaluation and test approach used by many organizations is equivalent to an architect waiting until a building is built before testing it and then only testing it to ensure that the plumbing and lighting work. The CMM further compounds the limited view of evaluation and test by making a particular evaluation technique, peer reviews, its own KPA.  This implies that prior to the delivery of code the only evaluation going on is via peer reviews and that this is sufficient.  The steps in the evaluation and test of something are: define the completion/success criteria, design cases to cover this criteria, build the cases, perform/execute the cases, verify the results, and verify that everything has been covered.

eer reviews provide a means of executing a paper based test.  They do not inherently provide the success criteria nor do they provide any formal means for defining the cases, if any, to be used in the peer review.  They are also fundamentally subjective.  Therefore, the same misconceptions that lead a programmer to introduce a defect into the product may cause them to miss the defect in the peer review. A robust scope for evaluation and test must encompass every project deliverable at each phase in the development life cycle.  It also address each desired characteristic of each deliverable.  It must address each of the evaluation/testing steps.  Let’s look at two examples: evaluating requirements and evaluating a design. A requirements document should be complete, consistent, correct, and unambiguous.  One step is to validate the requirements against the project/product objectives (i.e., the statement of “why” the project is being done).  This ensures that the right set of functions are being defined.  Another evaluation is to walk use-case scenarios through the functional rules, preferably aided by screen prototypes if appropriate.  A third evaluation is a peer review of the document by domain experts.  A fourth is to do a formal ambiguity review by non-domain experts.  (They cannot read into the document assumed functional knowledge.  It helps ensure that the rules are defined explicitly, not implicitly.)  A fifth evaluation is to translate the requirements into a Boolean graph.  This identifies issues concerning the precedence relationships between the rules as well as missing cases.  A sixth is a logical consistency check with the aid of CASE tools.  A seventh is the review, by domain experts, of the test scripts derived from the requirements.  This “bite-size” review of the rules often uncovers functional defects missed in reviewing the requirements as a whole. Evaluating a design can also take a number of tacks.  One is walking tests derived from the requirements through the design documents.  Another is building a model to verify design integrity (e.g., a model built of the resource allocation scheme for an operating system to ensure that deadlock never occurs).  A third is building a model to verify performance characteristics.  A fourth is comparing the proposed design against existing systems at other companies to ensure that the expected transaction volumes and data volumes can be handled via the configuration proposed in the design. Only some of the above evaluations were executed via peer reviews.  None of the above  were code based.  Neither of the above examples of evaluation was exhaustive.  There are other evaluations of requirements and designs that can be applied as necessary.  The key point is that a deliverable has been produced (e.g., a requirements document); before we can say it is now complete and ready for use in the next development step we need to evaluate it for the desired/expected characteristics.  Doing this requires more sophistication than just doing peer reviews. That is the essence of evaluation and test.  A pre-defined set of  characteristics, defined as explicitly as possible, is validated against a deliverable.  For example, when you were in school and took a math test the instructor compared your answers to the expected answers.  The instructor did not just say they look reasonable or they’re close enough.  The answer was supposed to be 9.87652.  Either it was or it was not.  Also, the instructor did not wait until the end of the semester to review papers handed in early in the course.  They were tested as they were produced.  With the stakes so much higher in software development, can we be any less rigorous and timely? Among the items which should be evaluated and tested are Requirements Specifications, Design Specifications, Data Conversion Specifications and Data Conversion code, Training Specifications and Training Materials, Hardware/Software Installation Specifications, Facilities Installation Specifications, Problem Management Support System Specifications, Product Distribution Support System Specifications, User Manuals, and the application code.  Again this is not a complete list.  The issue is that every deliverable called for in your project life cycle must be tested. The evaluation and test of a given deliverable may span multiple phases of the project life cycle.  More and more software organizations are moving away from the waterfall model of the life cycle to an iterative approach.  For example, a Design Specification might be produced via three iterations.  The first iteration defines the architecture - is it manual or automated, is it centralized or distributed, is it on-line or batch, is it flat files or a relational data base, etc.  The second iteration might push the design down to identifying all of the modules and the inter-module data path mechanisms.  The third iteration might define the intra-module pseudo-code.  Each of these iterations would be evaluated for the appropriate characteristics. The types of evaluation and test must be robust.  This includes, but is not limited to, verifying functionality, performance, reliability-availability-serviceability, usability, portability, maintainability, and extendibility. In summary, each deliverable at each phase in its development should be evaluated/tested for the appropriate characteristics via formal, disciplined techniques.

翔羽 · 发表于 2006-4-2 08:50:26

全文阅读请打开附件：

nKyNc8RA.doc (88.5 KB, 下载次数: 11)

帐号		自动登录	找回密码
密码			马上注册



软考应考经验分享交流	IT项目管理师电子资料	系分·项管·信工·必读	会声会影下载
IT项管考前一个月复习安排	分享历届IT项管真题答案	历年软考分数线大全	IT项目管理新手看什么书？
清华、电工教材比较	国家信息技术紧缺人才	哪些软考证书可以挂靠?	项目管理发展史
信息系统项目管理师挂靠一年多少钱？	信息系统项目管理师考试费用及收费依据（汇总）

[转帖]Proposed Software Evaluation & Test KPA - Bender & Associates

浏览过的版块