Product Ideas Portal

Violation status history for regression/validation testing

Issue:

Currently, when a client requests validation/regression testing we have to copy the report and write notes manually for "not fixed" and "partially fixed". If an issue is marked as fixed some clients want a separate report with all the fixed issues removed so the compliance score updates. If there are multiple rounds of testing, there could many comments that are too long to fit the "code snippet" text field with the character limit of 3000. Having to create multiple reports is time consuming and does not bode will when the client would like the report(s) immediately. Additionally, it can take longer than necessary to edit out fixed issue for VPATs.

 

Suggested solution (so there is no need to make copies of reports) :

1. Testers can provide violation status (Fixed, Partially Fixed, Not Fixed). When a violation is Fixed, it no longer gets counted against the Compliance Score for clients and does not show up in the Compliance Tables for VPAT creation.

2. Clients can filter out the Fixed violations in the module view.

3. Clients can see Compliance Score history in the Dashboard and Violation Status history (including tester notes) in module views and other violation views.
 

  • Amy Li
  • Apr 4 2019
  • Future consideration
AMP
  • Attach files
  • Jim Griesemer commented
    10 Apr 17:57

    I like this idea! I think it should include the ability to identify new defects that have appeared from real regression testing. It's not uncommon to validate that a fix has been made, but the process three new issue have been created.

  • Guest commented
    12 Apr 13:15

    Great idea! Plus, if everything is housed in one report, you can easily see their compliance score change over time - maybe a line graph?

  • Mitchell Evan commented
    17 Apr 17:25

    +1, thank you! Violation Status should be its own field, and fixed issues should improve the report's compliance score.

    When solving this enhancement request, please keep in mind these related business needs...

    • In addition to Violation Status, consultants should also provide retest details such as a prose comment (mentioned in the original idea as "tester notes", e.g. describing a partial fix) and the date of retesting. These details might or might not deserve their own dedicated fields. We'll need some method to continue providing this information which we currently provide in most cases.
    • When retesting a pattern, should we report status and retest details (A) once for the pattern as a whole, (B) once for each violation in the pattern, or (C) separately for each violation instance in each affected module? I recommend (B), which covers most retesting scenarios. For the fairly common case of two or more pattern violations deserving the same status and retest details, consider also offering (A) as an efficiency optimization for consultants.
    • When retesting a pattern, I sometimes find that all of the pattern's violations have been fixed in some modules, while in other modules none of the pattern's violations have been fixed. I can currently uncheck the modules where the pattern was fully fixed. This is efficient for AS consultants and it improves the compliance score. I recommend that we keep this the same functionality, and train consultants to select "Partially Fixed" Violation Status and list the fixed modules in a comment. As an efficiency optimization, consider a function to record the history of fixed modules automatically.
    • Sometimes when I retest, I find that a client's website has changed its behavior in a way that requires a rewrite of the violation instance or pattern. Please keep this less common case in mind, but I recommend not over-engineering this as part of the AMP/AST solution for retest status. When this happens, I recommend that the consultant create a new violation or pattern, and mark the original one "fixed" with a comment explaining that it was rewritten.
    • We often retest more than once. Bug databases handle this by overwriting status while keeping a log of all changes, which users can read in chronological order. The original idea mentioned "history", but I wonder if this might be better as a separate enhancement request to display a log of all changes (not just changes to Violation Status and retest details)? Either way, let's figure out what consultants should do in AMP/AST for multiple retests of a violation.

    These are my opinions. Product/Dev should please consult with AS users to confirm the solution will be efficient and meet business needs.