Apply W3 Test Methodology to ActivityPub
―bengo.is @ 2024-01-20
The Test Methodology
https://www.w3.org/TR/test-methodology/
- markup conformance requirements
- extract the conformance requirements from the spec
- establish a quality assurance process
- create test cases and testable assertions
- build at least one test case for each requirement
- test existing implementations to find bugs in test suite
markup conformance requirements
w3.org test-methodology says
Markup conformance requirements that need to be tested and give them a stable identifier that will persist across drafts of the specification.
ActivityPub Markup
ActivityPub has requirements marked up with rfc2119 CSS classes. Try document.querySelectorAll('.rfc2119') in the JS Console of that document.
extract the conformance requirements from the spec
w3.org test-methodology says
Extract the conformance requirements and examine them independently of the specification. Fix common mistakes and remove any duplicates.
activitypub-behaviors conformance requirements
The conformance requirements have been extracted from the spec and described in activitypub-behaviors, a dataset of behaviors described by the ActivityPub protocol. This includes the behaviors that are conformance requirements indicated by RFC2119 MUST.
activitypub-testing-website conformance requirements
activitypub-testing-website depends on activitypub-behaviors to render a webpage for each requirement, and an index at https://activitypub-testing-website.socialweb.coop/conformance/requirements/
establish a quality assurance process
w3.org test-methodology says
Establish a quality assurance process for both creating and verifying the test cases. In our case, this included providing a set of tools, templates, and methods explaining how to build useful test cases for the said specification (see [WIDGETS-PC-TESTS]). During the creation of the test suite for the [WIDGETS] specification, we also imposed a rule within those working on the test suite that a test case had to be independently verified before being committed into the final test suite. Although defective test cases still made it into the test suite, as more implementers worked their way through the test suite, the more bugs were found and fixed (both in the specification and in the test suite).
activitypub-testing QA
One process will be run by the activitypub-testing project, and will markup each test rule with a QA status as the test rules are impelmented, tested, critiqued, etc.
But there should in general be many concurrent communicating processes, never just one.
create test cases and testable assertions
w3.org test-methodology says
Create test cases and corresponding testable assertions. The act of converting prose into a computational form (a test case) can also help expose redundancies, ambiguities, and common mistakes. Bind the testable assertions to a conformance requirement via a stable identifier.
activitypub-tests and test rule expectations
Some test cases are in the activitypub-testing repository at /src/activitypub-tests/
We're iterating on a stable test rule API for defining test cases (inspired by Accessibility Conformance Testing Rules). Originally each test rule required one markdown file for humans and one corresponding javascript file for the test runner, and we're almost done making it only require a JavaScript file (but you'll still get an HTML page rendered from that on the website automatically). In the future there are ways it could only require writing a markdown/html file.
Within the test rule, the expectations are essentially testable assertions. See example expectations at https://activitypub-testing-website.socialweb.coop/test-cases/post-outbox-server-overwrites-id-property#expectations
build at least one test case for each requirement
w3.org test-methodology says
Build at least one test case for each conformance requirement. On average, our test suite contains 3 test cases per conformance requirement, with some assertions having 10 or more test cases.
Tracking this goal at https://activitypub-testing-website.socialweb.coop/conformance/requirements/tests/
test existing implementations to find bugs in test suite
w3.org test-methodology says
Compare the results of running these test cases on existing implementations to find bugs in the specification or in the test suite. For example, if one finds that all implementations are failing a test case, it might mean that the test case is defective.
See Testing an Actor at onepage.pub.
I encourage anyone to use activitypub-testing on their implementation and then post on the social web about how it goes. Then we will collectively be doing this part of the methodology.
Reflecting on Applying the Test Methodology So Far
A couple months in, it seems like a worthwhile methodology to apply if only to make the more complex testing activity more legible to certain audiences.
While there is some pretense and prerequisites required to be familiar with the whole thing, I have found it worthwhile to research those.
As always, reusing or even vaguely aligning with prior art and/or standards hopefully makes the work of activitypub-testing more accessible, maintainable, and reviewable by the broader w3 community.
- tags