A test case is a file or set of files that demonstrate how markup in a specific technology may pass or fail with regard to a WCAG checkpoint (WCAG 1.0) or success criterion (WCAG 2.0).
Formal metadata are "administrative" information (as opposed to semantic metadata).
A summary of the test materials and how they are to be used in the test case. The description indicates to the accessibility expert/scenario author what to expect and what will happen when a user interacts with the test materials. The description of the behaviour is especially important for test materials with interactive components such as forms.
A title for the test case. It does not need to be unique and its lenght can vary depending on the complexity of the test materials.
Date in ISO 8601 format: yyyy-mm-dd. Note: the restriction (start date and end date of BenToWeb) was added for internal use only, but should be removed when TCDL is submitted to W3C or elsewhere.
One of an enumerated list of values that can be used to track the state of a test case at a given time. See Status in W3C Working Group Note on Test Metadata.
An identifier that allows one to distinguish between different revisions of a test case. See Version in W3C Working Group Note on Test Metadata.
The source element must be used if the test case is borrowed from an existing test suite.
The test suite from which the test case is borrowed.
The name of the test suite. Use the name provided by the developers of the test suite.
The original file that was borrowed.
Indicate whether the borrowed file was modified before inclusion. Even a small change such as conversion from HTML 4.01 to XHTML 1.0 counts as a change.
General purpose container for comments on the borrowed materials, such as changes that were performed, errors that were found etcetera.
A description of the features of the file format of the test case.
The file format (proprietary, W3C recommendation or other) of the test case. There is one recommendation element for each file format used by the test case.
The name of the file format.
A description of the elements or features used by the test case. This element can be omitted for binary formats (PDF, Flash, …).
A description of a specific element or feature. There is one testElement for each feature. For CSS, provide the name of the rule, selector, property, property value, … For proprietary elements and attributes (Microsoft's bgsound, Netscape's embed, etcetera), provide a link to the company's documentation.
Choose between specReference and specQuote. Using specReference is preferred.
The name of the element or feature.
The title of the relevant section in the specification. Use the title provided by the specification wherever possible; otherwise provide a succinct and meaningful title for the relevant section in the specification.
A quote from the relevant section of the specification. specQuote can be used if it is not possible to provide an exact pointer to the relevant section in the specification.
Describes what is necessary for the evaluation of the test case.
The intention of the test materials. The purpose contains an explanation of the expected evaluation result in regard to the relevant rules, checkpoints or success criteria. It does not need to repeat the rule or success criterion. (Strictly speaking, this goes beyond the Purpose in W3C Working Group Note on Test Metadata.)
Guidance to accessibility experts for evaluating the test case.
The validators who evaluate the test case, create scenarios and move the test case forward through the validation process.
The first validator in the validation process, who checks the work of the test case author.
The second validator in the validation process, who checks the work of the test case author and the first validator.
Conditions that must be met before this test is executed. See Preconditions in W3C Working Group Note on Test Metadata.
The test modes for the test case. Combinations of test modes can be defined by repeating the testMode element, each time with a different value.
However, this should be avoided.
Note that this is not the test mode for validating the test case for inclusion in the test suite, but the test mode that that is recommended to external users of the BenToWeb test suite when evaluating a test case.
The method for validating the test case (internal, not for publication).
Scenarios describe what combinations of technologies and experience a test case should be evaluated with. (Scenarios need not be defined for automatic testing.)
Container for any kind of notes on the scenario, for example, its purpose.
Additional guidance to the end user who evaluates the test case, especially with regard to the configuration of their user agent and/or assistive technology. The xml:lang attribute is required. The languages should match those used for questions.
One or more question that can be presented to a human evaluator (an end user) to evaluate a test case. There should be one question or set of questions per language represented in the evaluation panel. The xml:lang attribute is required.
A yes-no-question, with a third option (for options such as "Don't know", "Not applicable", etcetera).
(Provided for forward compatibility: if it should ever be necessary to provide a string other than 'Yes', this will need to be localised, hence the xml:lang attribute.)
The value submitted to the server if the user chooses 'Yes'. This value must be a positive integer if 'Yes' is the correct or expected answer, and a negative integer if 'Yes' is not correct or expected. Zero must not be used as a value.
(Provided for forward compatibility: if it should ever be necessary to provide a string other than 'No', this will need to be localised, hence the xml:lang attribute.)
The value submitted to the server if the user chooses 'No'. This value must be a positive integer if 'No' is the correct or expected answer, and a negative integer if 'No' is not correct or expected. Zero must not be used as a value.
Alternative for “yes” or “no”, for example “I don't know” or “Other”. If this element is used, there is one “other option” for each language.
A Likert scale is a type of survey question where respondents are asked to rate the level at which they agree or disagree with a given statement.
The label associated with the choice. (One per language).
The value submitted to the server; each level in a Likert scale or in a multiple choice question must have a unique value.
Numbering from 1 to 5.
Defines how a list of choices is displayed: vertically, horizontally or as a drop-down list. The default is vertical (for greater accessibility to screen readers). Dropdowns are not recommended if more than one choice needs to be selected.
A multiple choice question (strictly a generalisation of Likert scale). It is possible to limit the number of responses to one (for radio buttons) or to allow multiple reponses (for check boxes).
The label associated with the choice. (One per language).
The value submitted to the server. This value must be a positive integer for the correct or expected answer, and a negative integer for the incorrect or unexpected answers. Each integer (positive or negative) must be uniqe and must not be zero.
Numbering/order of the choices.
Defines whether the user can select one or multiple choices (i.e. one radio button versus multiple check boxes).
Defines how a list of choices is displayed: vertically, horizontally or as a drop-down list. The default is vertical (for greater accessibility to screen readers). Dropdowns are not recommended if more than one choice needs to be selected.
An open question, which the user can answer in a textarea.
Size of space on the form for the user's answer. Limiting the amount ot space can encourage respondents to prioritise their points. (Different languages have different space requirements; some need more space than English!)
The height of the text area (number of rows or lines).
The width of the text area (number of columns, or characters per line).
A yes-no-question, with a third option (for options such as "Don't know", "Not applicable", etcetera) and an open question.
(Provided for forward compatibility: if it should ever be necessary to provide a string other than 'Yes', this will need to be localised, hence the xml:lang attribute.)
The value submitted to the server if the user chooses 'Yes'. This value must be a positive integer if 'Yes' is the correct or expected answer, and a negative integer if 'Yes' is not correct or expected. Zero must not be used as a value.
(Provided for forward compatibility: if it should ever be necessary to provide a string other than 'No', this will need to be localised, hence the xml:lang attribute.)
The value submitted to the server if the user chooses 'No'. This value must be a positive integer if 'No' is the correct or expected answer, and a negative integer if 'No' is not correct or expected. Zero must not be used as a value.
Alternative for “yes” or “no”, for example “I don't know” or “Other”. There is one “other option” for each language.
Defines the space on the form, usually a textarea, that allows a user to explain why the third option was chosen instead of “yes” or “no”. Limiting the amount ot space can encourage respondents to prioritise their points.
The height of the text area (number of rows or lines).
The width of the text area (number of columns, or characters per line).
A user's level of experience with the specified technologies. There are three categories of technologies: assistive technologies, user agents and devices. User agent also covers browser plugins.
One or more disabilities (select from a list).
A short name that allows more user-friendly identification of scenarios then ID values.
The file or files that make up the test case. Usually there is only one file, but some checkpoints or success criteria, e.g. those on consistent navigation, consistent style etc, require a set of files. (In that case, we call the test case 'compound'.)
Language of the referenced resource (see HTML's hreflang attribute).
Indicates whether the order of the files (if there are several) matters for the evaluation of the test case. 'true' means that the files should be processed in the order in which they occur in the test case description. 'false' means that the files can be processed in random order. The default value is 'false'.
Captures whether the test case is expected to fail or passes relevant 'rules' (WCAG 1.0 checkpoints, WCAG 2.0 success criteria, Section 508 rules, etc.) IMPORTANT NOTE: We only specify whether the test case passes or fails the (WCAG 2.0) rule under consideration, so a 'pass' may still mean that the test cases fails with regard to other rules (see http://lists.w3.org/Archives/Public/w3c-wai-gl/2005AprJun/0117.html ).
A WCAG 1.0 checkpoint, WCAG 2.0 success criterion or Section 508 rule. In BenToWeb, we always refer to WCAG 2.0 success criteria; other 'rule sets' are strictly optional, so expected evaluation results for these other rules sets are purely auxiliary information.
The location or locations where relevant (especially faulty) code occurs.
The location of the code. Sometimes a combination of features (markup, rules, etc.) constitute an error when taken together (even if these features are not errors when considered separately). In that case, there should be one location element per feature/error. Use at least XPath or a combination of line and column number.
The file where a (potential) issue is located.
Line number where the relevant code occurs. Use in combination with column.
Column number where the relevant code occurs. Use in combination with line. Note: use the "column number where the current document event ends" (for compatibility with SAC).
XPath 1.0 expression, as defined in XML Path Language (XPath) 1.0 - W3C Recommendation 16 November 1999.
Value indicating whether the test case is epected to pass or fail with regard to the 'rule'. 'cannotTell' is used if there is ambiguity of unclarity (e.g. dependence on the implementation or version of a user agent or assistive technology, ambiguity in the accessibilty rules or in the specification of the file format).
A description of why the test case passes or fails in terms that are relevant to a user's experience (as apposed to technical comments on source code).
A technical description of why the test case passes or fails.
If more than one rule is listed, this attribute indicates whether a rule is the primary rule or an additional, informational rule. Primary rules have and AND relationship.
This is a pointer to a checkpoint (WCAG 1.0), success criterion (WCAG 2.0), rule (Section 508) etc described in ruleset.xml.
Namespace prefix for the XPath expression. This attribute is required when an XPath expression is used. Note: the namespace prefix can be an empty string.
Namespace URI for the XPath expression. This attribute is required when an XPath expression is used.
A unique identifier for the test case description. The identifier must be unique in the test suite in which the test case belongs. The following is an example of a convention that ca be used in a test suite for WCAG 2.0: technology_wcag2_date_x.y_lz_scz_nnn, where
The ID for a XHTML test case for guideline 1.1 level 1 success criterion 1 may look like this: xhtml1_wcag2_20041119_1.1_l1_sc1_001.
A test case is atomic if it applies to only one checkpoint (WCAG 1.0) or success criterion (WCAG 2.0), or complex if it applies to multiple checkpoints or success criteria.
The test mode(s) required for this test case. The choices are:
Note: in BenToWeb, expert evaluation happens by default (in order to decide whether a test case needs to be evaluated by end user, automatically).
Note that there is no value for combinations of test modes: combinations can only be defined by repeating the element that specifies the test mode.
A scale from 1 (low/novice) to 5 (high/expert) that indicates a user's (required) experience.
HTTP method (not all allowed methods are relevant to test suites, namely 'TRACE', 'OPTIONS', 'DELETE', 'CONNECT').
MIME types @TODO complete [see 'encoding' below and list of MIME types at IANA ].
Encoding types (e.g. for use in forms submission with HTML form [enctype] and with XForms. @TODO complete. [refer to 'media types': RFC 2045: 'Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies']
Evaluation result expected by the developer of the test case.
Mixed content (for use in labels, names, comments, …).
HTML-style paragraph, but greatly restricted.
Question presented to the end user. (One per language.)
A label for a form. (One per language.)
Any software that retrieves and renders Web content for users. This may include Web browsers, media players, plug-ins, and other programs — including assistive technologies — that help in retrieving and rendering Web content. Note: in BenToWeb, assistive technologies are not user agents but are a category of their own.
Assistive technologies such as screen readers, screen magnifiers, voice recognition software, alternative keyboards and alternative pointing devices.
A category of assistive technology.
A specific product by a manufacturer of assistive technology.
The hardware on which user agents and/or assistive technologies run: personal computers, personal digital assistants, mobile phones, …
A disability or impairment. The choices are limited to those that are relevant to accessibility evaluation; they are not based on the WHO's International Classification of Functioning, Disability and Health.
See Status in W3C Working Group Note on Test Metadata.
A test case goes through the following process (simplified version):
Specifies whether something is inside or outside the baseline. The value 'included' means that something is inside the baseline, 'excluded' means outside the baseline. (For information about baselines in WCAG 2.0, see Technology assumptions and the "baseline" in Web Content Accessibility Guidelines 2.0 (W3C Working Draft 27 April 2006) and About Baselines and WCAG 2.0.)