by bengo aka @bengo@mastodon.social
This post builds on a previous post Which ActivityPub Server Requirements are Easy to Test? which included a proposal:
Focus on the 'easy to test' ActivityPub server requirements and improve their testability.
(emphasis added)
That blog post focused on a way of determine which requirements are 'easy to test'. This blog post will focus on one way of improving the testability of a requirement.
"Testability" is often informally defined. What does it mean? How can we come to some kind of consensus on whether a requirement is testable or how to improve the testability of anything?
Standards communities have wrestled with this question over the years, and we can learn from those efforts.
The W3C QA Glossary defines Testability
Testability: A proposition is testable if there is such a procedure that assesses the truth-value of a proposition with a high confidence level.
This definition, as with all words, leaves much open to interpretation, prompting more discussion and specification over time.
According to https://www.w3.org/wiki/TestableOrNot,
When designing a specification with testability or not, it may be hard to evaluate whether a given assertion is testable or not; testability is often confounded with objectivity (whereas subjective testing exists) or automatibility (whereas manual testing exists). So, what is testable? what's not?
According to https://www.w3.org/QA/WG/2002/07/qaframe-spec-0715,
The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable requirements and that can be used to easily generate Quality Assurance materials (tests and suites).
and
A specification is testable if there exists some finite cost-effective process with which a person or software can check that the requirement has been met.
This question is meant as a pause to brainstorm. If you have more ideas, please do share them.
If we accept a definition of testability like that in W3C QA Glossary that defines a proposition (e.g. a requirement) as testable "if there is such a procedure that assesses the truth-value of a proposition with a high confidence level", then we can improve testability for requirements by constructing a procedure that assesses the truth-value of a proposition that the requirement is met.
When a requirement has zero test procedures, we can improve testability by defining a single test procedure. But does it also improve testability to add a second test procedure for a requirement where a test procedure already exists? Yes.
We can improve testability not only by constructing a single procedure to test the proposition of a test target meeting the requirement, but also by constructing several procedures that can be used to test the requirement. Many test procedures per requirement lowers the cost of improving testability because then any one procedure doesn't have to perfectly test the requirement in all possible ways. There can be test procedures for the requirement that must only be performed if the required inputs for the procedure are available and meet applicability requirements for the procedure.
Test procedures can be composed. i.e. Test Procedure D can have result passed if and only if all of Test Procedures A, B, and C all have result passed.
How many test procedures should there be, and how many people should author them?
We will best reach wide-review and due process toward consensus by welcoming Test contributions from as many members as possible. Instead of having one document of tests edited by one person, we should have a catalog of test cases that anyone can propose an addition to.
One way that contributions to specifications can be discouraged is by not accepting draft submissions, i.e. by requiring all submissions to meet a bar of quality before being accepted. When these acceptance requirements are undocumented or unnecessarily high, this can lead to exclusion of valuable contributions.
Instead, a test submission format can include a way of indicating that the status of a submission is a 'draft'. i.e. that the author would like the draft added to the working version of the catalog, but intends to develop it further before seeking review or consensus.
To improve testability of ActivityPub requirements, we should publish Test Specifications that specify Test Procedures that test the proposition that a test target conforms to specific ActivityPub requirements.
A Test Specification may include
uuid - a unique identifier for the test specification that will not change over timetitle - a plaintext human-readable title of the test specificationstatus - e.g. 'draft', 'proposal', 'final'inputs - the inputs required to determine applicability and execute test procedures, e.g. a test subjectapplicability - describes the part of the test subject that is the test target of this testexpectations - describe what is required of the test targets in order to determine the test outcomeoutcomes - the outcomes of the test, e.g. Untested, Inapplicable, Passed, Failed, CantTellauthor - one or more authors of the test specificationA sketch of a Test Specification for one ActivityPub Test Requirement is https://socialweb.coop/activitypub/test-cases/actor-must-serve-as2-object-to-get/