Product Ideas Portal

Violation status history for regression/validation testing


Currently, when a client requests validation/regression testing we have to copy the report and write notes manually for "not fixed" and "partially fixed". If an issue is marked as fixed some clients want a separate report with all the fixed issues removed so the compliance score updates. If there are multiple rounds of testing, there could many comments that are too long to fit the "code snippet" text field with the character limit of 3000. Having to create multiple reports is time consuming and does not bode will when the client would like the report(s) immediately. Additionally, it can take longer than necessary to edit out fixed issue for VPATs.


Suggested solution (so there is no need to make copies of reports) :

1. Testers can provide violation status (Fixed, Partially Fixed, Not Fixed). When a violation is Fixed, it no longer gets counted against the Compliance Score for clients and does not show up in the Compliance Tables for VPAT creation.

2. Clients can filter out the Fixed violations in the module view.

3. Clients can see Compliance Score history in the Dashboard and Violation Status history (including tester notes) in module views and other violation views.

  • Amy Li
  • Apr 4 2019
  • Future consideration
  • Attach files
  • Admin
    Zahra Safavian commented
    27 Nov, 2019 08:17pm

    A user let us know that having the defect status in the VPAT helps in tracking issues. The user stated,

    "When I retest the site for regression, I use the defect status to update the state of the violation in the original report. This helps us keep our issue status consistent with our bug tracking system.

    A little background on our testing process might help.

    Being in an agile environment, we update issues per sprint which means we are not “retesting” the entire site. This could create new issues as well as remove issues which are fixed.

    We can be requested to create a VPAT for the application at given point, and hence it is important for us to have the entire picture in one report rather than the “workaround” of multiple reports.

    Additionally, the defect status really helps us sync with the defect in our bug tracking system.”

  • Mitchell Evan commented
    17 Apr, 2019 05:25pm

    +1, thank you! Violation Status should be its own field, and fixed issues should improve the report's compliance score.

    When solving this enhancement request, please keep in mind these related business needs...

    • In addition to Violation Status, consultants should also provide retest details such as a prose comment (mentioned in the original idea as "tester notes", e.g. describing a partial fix) and the date of retesting. These details might or might not deserve their own dedicated fields. We'll need some method to continue providing this information which we currently provide in most cases.
    • When retesting a pattern, should we report status and retest details (A) once for the pattern as a whole, (B) once for each violation in the pattern, or (C) separately for each violation instance in each affected module? I recommend (B), which covers most retesting scenarios. For the fairly common case of two or more pattern violations deserving the same status and retest details, consider also offering (A) as an efficiency optimization for consultants.
    • When retesting a pattern, I sometimes find that all of the pattern's violations have been fixed in some modules, while in other modules none of the pattern's violations have been fixed. I can currently uncheck the modules where the pattern was fully fixed. This is efficient for AS consultants and it improves the compliance score. I recommend that we keep this the same functionality, and train consultants to select "Partially Fixed" Violation Status and list the fixed modules in a comment. As an efficiency optimization, consider a function to record the history of fixed modules automatically.
    • Sometimes when I retest, I find that a client's website has changed its behavior in a way that requires a rewrite of the violation instance or pattern. Please keep this less common case in mind, but I recommend not over-engineering this as part of the AMP/AST solution for retest status. When this happens, I recommend that the consultant create a new violation or pattern, and mark the original one "fixed" with a comment explaining that it was rewritten.
    • We often retest more than once. Bug databases handle this by overwriting status while keeping a log of all changes, which users can read in chronological order. The original idea mentioned "history", but I wonder if this might be better as a separate enhancement request to display a log of all changes (not just changes to Violation Status and retest details)? Either way, let's figure out what consultants should do in AMP/AST for multiple retests of a violation.

    These are my opinions. Product/Dev should please consult with AS users to confirm the solution will be efficient and meet business needs.

  • Guest commented
    12 Apr, 2019 01:15pm

    Great idea! Plus, if everything is housed in one report, you can easily see their compliance score change over time - maybe a line graph?

  • Jim Griesemer commented
    10 Apr, 2019 05:57pm

    I like this idea! I think it should include the ability to identify new defects that have appeared from real regression testing. It's not uncommon to validate that a fix has been made, but the process three new issue have been created.