A useful software review does two jobs in the right order: it rejects options that fail a mandatory condition, then compares the remaining products using evidence. A single feature score cannot safely combine workflow fit, security, implementation effort and exit risk.
The 15 criteria below are organised into five lenses. For each criterion, the review sheet records whether it is a gate or a scored preference, what proof was observed, what remains uncertain and who owns the next check. This turns a review from an opinion about a demonstration into an auditable buying decision.
Five lenses prevent an attractive feature set from hiding a failed control, operating constraint or exit requirement.
Before scoring: decide what can disqualify a product
Some requirements are conditions of entry. If a system cannot complete a critical workflow, meet a mandatory control, operate within the available environment or stay inside an affordable range, its strengths elsewhere do not repair the failure. Mark these items pass, fail or unresolved before calculating any weighted score.
Everything else may be compared using an anchored scale. A simple four-point scale is usually easier to defend than percentages:
- 0 — unacceptable or no evidence;
- 1 — material gap or dependence on uncommitted future work;
- 2 — acceptable with known effort or limitation;
- 3 — strong fit demonstrated with relevant evidence.
Set the gates, weights and evidence standard before supplier demonstrations. Otherwise a team can unconsciously change the method to favour the product it already likes.
Lens 1: does the software fit the business?
1. Outcome and workflow fit
Start with the jobs the system must support and the change the organisation expects. Ask finalists to complete the same representative workflows, including at least one exception. A generic tour proves that screens exist; it does not prove that an order correction, returned approval, disputed invoice or reassigned case works in your context.
Evidence: scripted demonstration or pilot using representative roles and data, with elapsed time, hand-offs and workarounds recorded. Warning sign: requirements are feature names copied from product pages rather than statements about the work.
2. User and accessibility fit
Review fit by role, frequency and working environment. A daily specialist, occasional approver, mobile field worker and external participant face different friction. Include keyboard operation, readable focus states, zoom, contrast, error recovery and assistive-technology needs where relevant. Accessibility is not adequately tested by asking whether a product is “compliant”; identify the applicable standard and verify the workflows your users perform. WCAG 2.2 is a current W3C Recommendation for web-content accessibility, but the buyer still needs evidence for the product, scope and user context being evaluated.
Evidence: role-based task tests and accessibility evidence with scope. Warning sign: the easiest user journey is treated as representative of everyone.
3. Scale and change fit
Test the likely next operating state, not an abstract promise that the software “scales.” Consider data volume, peak concurrency, new locations, legal entities, languages, approval depth and product configuration as the organisation changes. Also ask what becomes harder: administration, reporting, permissions, integration or price.
Evidence: documented limits, relevant reference architecture or a controlled volume/configuration test. Warning sign: scale is discussed only as infrastructure capacity while operating complexity is ignored.
Lens 2: what can the product actually demonstrate?
4. Core workflow capability
Distinguish standard capability, configuration, custom code, third-party dependency and roadmap promise. These paths have different costs and risks even if the demonstration looks identical. Record which path was used for each critical step and who would own it after launch.
Evidence: live scenario completion in a product version and configuration relevant to the proposal. Warning sign: slides, prototypes or a partner tool are scored as currently available standard behavior.
5. Data and reporting
Review the data lifecycle: entry, validation, ownership, correction, history, retention, reporting, export and deletion. Create one required operational report and one management question from representative data. Confirm whether users can trace a number back to the records and definitions that produced it.
Evidence: import/export samples, data dictionary, lineage or audit view, and a report built from buyer-defined fields. Warning sign: “unlimited reporting” is accepted without testing definitions, permissions or extractability.
6. Integration and standards
List the events and data that must cross system boundaries, then evaluate the mechanism: supported connector, documented API, file transfer, event/webhook, middleware or manual work. Check authentication, limits, error handling, monitoring, versioning and ownership—not only whether an integration logo appears on a marketplace page. Open standards can reduce dependence and make future change easier; the GOV.UK open-standards guidance highlights interoperability, reuse and reduced lock-in as benefits.
Evidence: interface documentation and a test that includes a failed or duplicate transaction. Warning sign: integration effort is excluded from both the delivery plan and cost model.
Lens 3: can the organisation trust and control it?
7. Security assurance
Define the security outcomes that matter for the proposed use: identity, least privilege, secure defaults, vulnerability handling, audit, incident response and supplier practices. Ask what the evidence covers and how current it is. CISA's Secure by Demand Guide recommends that buyers integrate product-security questions before, during and after procurement rather than treat security as a one-time checklist.
Evidence: scoped independent assurance, product documentation, contractual commitments and buyer tests proportionate to risk. Warning sign: an old certificate or company-level badge is assumed to cover the exact product and service.
8. Privacy and information governance
Map which personal or sensitive data enters the system, why it is needed, where it is processed, who receives it and how it is retained or removed. Review sub-processors, cross-border arrangements, data-subject or records-management obligations, and the buyer's ability to configure controls. Obtain jurisdiction- and sector-specific advice where required.
Evidence: processing terms, data-flow information, retention/deletion behavior and governance configuration. Warning sign: the privacy policy is treated as a substitute for understanding the proposed processing.
9. Resilience and recovery
Availability percentages are only one part of resilience. Ask how incidents are detected and communicated, how data is protected, what restoration objectives apply, how dependencies affect service and what the buyer must do. Test the operational response: who works differently when the system or an integration is unavailable?
Evidence: service objectives, incident history where available, recovery evidence, status communications and buyer continuity procedure. Warning sign: the contract gives a service credit but the business has no workable recovery path.
Lens 4: what will it take to operate the system?
10. Implementation effort
Estimate discovery, configuration, data preparation, migration, integration, testing, training, change work and transition support. Identify dependencies on supplier specialists or partners. Separate a credible implementation plan from a sales estimate by naming deliverables, owners, acceptance criteria and assumptions.
Evidence: work breakdown, responsibility matrix, acceptance plan and effort range. Warning sign: substantial work is described as “configuration” and therefore given no owner or budget.
11. Administration and support
List recurring tasks: user and permission management, configuration, release review, data quality, integrations, reporting, audit requests and support escalation. Estimate the skills and time required internally. Review support channels, hours, severity definitions and what counts as chargeable consulting.
Evidence: admin task demonstration, support policy, release process and named operating model. Warning sign: the product is described as maintenance-free because the supplier hosts it.
12. Adoption and usability
Adoption is not a communication campaign applied after selection. Test whether the system reduces or adds steps, whether occasional users can recover after time away, and whether critical work still needs spreadsheets or messaging outside the product. Define adoption by completed behavior, not login count.
Evidence: representative task tests, observed friction, training effort and a measure such as proportion of target transactions completed correctly in the system. Warning sign: low adoption is assumed to be user resistance before workflow or configuration problems are examined.
Lens 5: is the commitment commercially resilient?
13. Total cost of ownership
Compare one-time, recurring and change-driven costs across the same time horizon. Include licences or usage, implementation, migration, integrations, internal administration, support, training, growth, customisation maintenance and exit. Record assumptions separately from confirmed prices.
Evidence: multi-year cost model with volume and effort ranges. Warning sign: the cheapest licence is declared the cheapest option before operating and exit costs are known.
14. Contract and supplier fit
Review whether the agreement reflects the implementation and operating promises on which the decision depends. Examine responsibilities, acceptance, service, renewal, price changes, data terms, support, liability, termination and transition. Supplier viability and strategic fit matter, but do not convert size or brand recognition into proof that your workflow will be supported.
Evidence: marked contract, risk log and written commitments tied to owners. Warning sign: decisive promises remain only in sales correspondence or roadmap discussions.
15. Exit and portability
Plan the end before signing. Define the export formats, relationships, attachments, audit history and configuration information needed to move or archive the service. Ask how long retrieval takes, whether APIs remain available during transition, what assistance costs and when remaining data is deleted.
Evidence: sample export, documented termination process and transition terms. Warning sign: “you own your data” is accepted without proving that usable data can leave.
The review sheet: one row per claim, not one score per product
Build the working sheet with these columns:
| Field | Purpose |
|---|---|
| Criterion and specific requirement | Prevents broad labels from hiding what was actually tested |
| Gate or weighted preference | Stops optional strengths compensating for a mandatory failure |
| Evidence and date | Links the judgment to an observation or document |
| Status or anchored score | Makes the interpretation consistent |
| Uncertainty | Keeps missing proof visible |
| Owner and next action | Turns an open point into accountable work |
After all mandatory gates pass, calculate the weighted comparison and run a sensitivity test. Change the most debatable weights within a reasonable range. If the preferred product changes repeatedly, the result is not a clear winner; obtain better evidence or ask the decision maker to accept the trade-off explicitly.
What a completed review should make obvious
A reader who did not attend the demonstrations should be able to see why every finalist passed or failed, which evidence supports the scores, where uncertainty remains, what implementation will demand and what would make the organisation reconsider the choice. That is a stronger review than a long narrative, a feature count or a single number with false precision.
Use the 15 criteria as coverage, not as a universal weighting. The organisation's workflows, data, risk and operating model decide which criteria are gates and which deserve weight. The method is successful when it exposes the decision—not when it produces the most elaborate spreadsheet.