🎁 activitypub-testing: a test runner for ActivityPub Actors

bengo @ 2024-01-01T20:38:05

A little over a month ago, as I prepared for this festive season, where we gather with loved ones and demonstrate generosity and reflect on where we've been over the last year …and where we need to go …and how we'll know it if we get there …and what to do if we do, I thought about what I wished to receive as gifts, and what I wished to gift to others.

So I made a wishlist. Not just any wishlist. a Testing Ecosystem Wishlist.

And while I know I was naughty this year, I must have been nice too.

Wonka: "Don't forget what happened to the man who suddenly got everything he wanted."
Charlie Bucket: "What happened?"
Mr. Wonka: "He lived happily ever after."

― Willy Wonka, Roald Dahl

Well I didn't get everything I wanted. But I have some good gifts to share.

One of the gifts I wished for is for us all to be ok with not gifting more.

Testing Wishlist Round 2: Which Wishes Came True?

Let's color the list. green for what I got. red for what not.

As a

end-user

I can see test assertions about all the service providers I use so I know which ones may not work as expected

I can see test assertions about service providers I am evaluating for future usage so I can choose one that will reliably interoperate and which will maximally provide end-user benefits afforded for by the protocol specification

test operator e.g. a protocol implementor testing their implementation

I can list all the tests that are available to me

I can inspect a single test

I can run an automated test to determine the result for a test subject

I can run all the automated tests that are appropriate for a provided test subject and see the results

I can easily explore the test results

I can show off my awesome test results

test author

I can add a test case

I can give examples of applying my test case

I can solicit feedback on my test case so I can make it better

I can discover ideas for tests that need to be newly authored or improved

service provider

I can discover the results of tests about the service I am responsible for

I can easily explore the test results

I can get advice on actions I can take to improve the results of tests run on my system

I can run several testing tools and merge the results

I can try out many testing tools and am never locked into just one

Let the Gifts Through 🎁

activitypub-behaviors 🎁

a dataset of behaviors described by the ActivityPub protocol.

― activitypub-behaviors/README

activitypub-behaviors contains a Conformance Requirement Dataset for ActivityPub.

activitypub-testing 🎁

tools for testing implementations of ActivityPub, a decentralized social networking protocol.

― activitypub-testing/README

activitypub-testing contains a suite of Test Cases for ActivityPub, both human readable and automated, and provides tools for running these tests against ActivityPub Actors.

`activitypub-testing/test-cases`

There are conformance tests for each of the 11 easy to test ActivityPub Requirements I described at the 2023-10-18 w3.org SocialCG Testing Task Force Meeting, and we'll continue until there are tests for all ~50 of the required activitypub-behaviors.

Test Case development has been inspired by A Method for Writing Testable Conformance Requirements from the W3C Mobile Web Test Suites Working Group.

diagram of w3.org test-methodology

The Test Cases we've written so far were heavily inspired by the Accessibility Conformance Testing (ACT) Rules Format 1.0 from the W3C Accessibility Guidelines WG.

`activitypub-testing` Command Line Interface

a.k.a. the activitypub-testing cli

If you have npm installed, you can install the cli with npm install -g activitypub-testing.

Running the activitypub-testing cli with no arguments shows docopt documentation on how to use it:

⚡ activitypub-testing

activitypub-testing

What?
  activitypub-testing is a cli for testing implementations of ActivityPub[0]

Usage:
  # browse all the tests
  activitypub-testing get tests [--output=<mediaType>]

  # inspect a single test
  activitypub-testing get test (--uuid=<uuid>|--slug=<slug>) [--output=<mediaType>]

  # run a test
  activitypub-testing run test (--uuid=<uuid>|--slug=<slug>) --input.anyInputName=<input.anyInputName>

  # run tests on a specific ActivityPub Actor
  activitypub-testing test actor <actor-uri> [--output=<mediaType>]

  # print this help
  activitypub-testing [--help|-h] [help]

Options:
  -h --help        Show this help text.
  -o --output      Choose output media type (default `text`, also allows `json`, `yaml`)
  --slug           slug (aka URL-path-friendly human-readable name) of selection
  --uuid           UUID (rfc4122) identifier of selection

Examples:
  # runs test inbox-must-be-an-orderedcollection[1] against a valid actor fetched via https
  activitypub-testing \
    test \
    --uuid '5e94d155-ed4a-4d71-b797-d7c387736ecf' \
    --input.object="$(curl -s 'https://socialweb.coop/activitypub/actors/with-empty-inbox.json')"

  # run tests on a specific ActivityPub Actor (pipe to `jq` to pretty print JSON)
  activitypub-testing \
    test actor \
    https://socialweb.coop

[0]: https://en.wikipedia.org/wiki/ActivityPub
[1]: https://socialweb.coop/activitypub/test-cases/inbox-must-be-an-orderedcollection/

For more sample invocations, check out the README

`activitypub-testing` for Continuous Delivery of ActivityPub Servers

The primary use case for this is for developers to run tests from the command line, either while developing the application, or by adding activitypub-testing invocations to their Test Automations to run ActivityPub tests as part of Continuous Integration and Continuous Testing pipelines.

According to Carnegie Mellon University's Software Engineering Institute, "continuous testing" is Stage 3 in A Framework for DevSecOps Evolution and Achieving Continuous-Integration/Continuous-Delivery (CI/CD) Capabilities and enables Continuous Delivery in Stage 4.

The benefits of CI/CD are further described in nist.gov SP 800-204: Security Strategies for Microservices-based Application Systems. and nist.gov SP 800-215: Guide to a Secure Enterprise Network Landscape.

`activitypub-testing/website`

an open source website to render the contents of the activitypub-testing package

― activitypub-testing/website/README

The tests in activitypub-testing are pretty useful through the cli, but they can also be used as a JavaScript library, e.g. to make a website that renders all the requirements and/or test cases. activitypub-testing/website shows how to use activitypub-testing as a library to make a website (built by 11ty).

The website is deployed at activitypub-testing.socialweb.coop for you to try it out, but you should probably dive into the source code repository to see it how it works.

It also includes an Implementation Guide for how to make use of activitypub-testing in your project.

activitypub-actor-tester 🎁

an HTML element that renders a tool for testing an ActivityPub Actor using activitypub-testing/test-actor

― activitypub-actor-tester/README

Any web page can be an ActivityPub Actor testing tool by adding an <activitypub-actor-tester> element and including the custom element definition in a script tag, eg.

<script webc:keep
  src="https://cdn.jsdelivr.net/npm/activitypub-actor-tester@0.2.2/dist/activitypub-actor-tester.js"
  integrity="sha384-LA7dbf/ovh5DDtqSW+DNFa7Z099LvPbl9pW6+Mb0wXYmH6QAMdDNseKglnSQcYST"
  type="module"
  crossorigin="anonymous"
></script>

To prove this, I'll do so right here. Try entering an Actor URL like https://bengo.is/actor.json and click 'Test'. Note, however, that the requests from the widget must be authorized by CORS headers from the ActivityPub server.

We'll add tests and improve the UX of this element over time. And you're also welcome to branch off in the repository and hack it however you want to make your own ActivityPub testing elements. Check out how it works in activitypub-actor-tester.js

With these Gifts, What can we do?

I can see test assertions about all the service providers I use so I know which ones may not work as expected

I use mastodon.social as an ActivityPub Server so others can follow me on the social web at https://mastodon.social/@bengo.

But sometimes I wonder whether other non-mastodon implementations will be able to interoperate with my Actor there.

⚡ activitypub-testing test actor https://mastodon.social/@bengo | jq .

output

I can list all the tests that are available to me

⚡ activitypub-testing get tests

show output

name: ActivityPub Tests
type:
  - Collection
items:
  - slug: actor-objects-must-have-inbox-outbox-properties
    uuid: acaacb5f-8f7e-4f28-8d81-c7955070a767
    url: https://socialweb.coop/activitypub/test-cases/acaacb5f-8f7e-4f28-8d81-c7955070a767
    id: urn:uuid:acaacb5f-8f7e-4f28-8d81-c7955070a767
  - slug: actor-must-serve-as2-object-to-get
    uuid: e7ee491d-88d7-4e67-80c8-f74781bb247c
    url: https://socialweb.coop/activitypub/test-cases/e7ee491d-88d7-4e67-80c8-f74781bb247c
    id: urn:uuid:e7ee491d-88d7-4e67-80c8-f74781bb247c
  - slug: inbox-must-be-an-orderedcollection
    uuid: 5e94d155-ed4a-4d71-b797-d7c387736ecf
    url: https://socialweb.coop/activitypub/test-cases/5e94d155-ed4a-4d71-b797-d7c387736ecf
    id: urn:uuid:5e94d155-ed4a-4d71-b797-d7c387736ecf
  - slug: outbox-must-be-an-orderedcollection
    uuid: 4af549f4-3797-4d99-a151-67c3d8feaa46
    url: https://socialweb.coop/activitypub/test-cases/4af549f4-3797-4d99-a151-67c3d8feaa46
    id: urn:uuid:4af549f4-3797-4d99-a151-67c3d8feaa46
  - slug: shares-collection-must-be-a-collection
    uuid: b03a5245-1072-426d-91b3-a3d412d45ae8
    url: https://socialweb.coop/activitypub/test-cases/b03a5245-1072-426d-91b3-a3d412d45ae8
    id: urn:uuid:b03a5245-1072-426d-91b3-a3d412d45ae8
  - slug: likes-collection-must-be-a-collection
    uuid: 200b9bc8-aae3-46f2-a6ab-5366042c0f6e
    url: https://socialweb.coop/activitypub/test-cases/200b9bc8-aae3-46f2-a6ab-5366042c0f6e
    id: urn:uuid:200b9bc8-aae3-46f2-a6ab-5366042c0f6e
  - slug: liked-collection-must-be-a-collection
    uuid: 018c3df2-d6d8-7f62-805b-b71a96cc6170
    url: https://socialweb.coop/activitypub/test-cases/018c3df2-d6d8-7f62-805b-b71a96cc6170
    id: urn:uuid:018c3df2-d6d8-7f62-805b-b71a96cc6170
  - slug: followers-collection-must-be-a-collection
    uuid: 018c3e08-611f-7e56-9f45-2fe5e4877d4e
    url: https://socialweb.coop/activitypub/test-cases/018c3e08-611f-7e56-9f45-2fe5e4877d4e
    id: urn:uuid:018c3e08-611f-7e56-9f45-2fe5e4877d4e
  - slug: following-collection-must-be-a-collection
    uuid: 018c3e17-a1bd-7040-8007-4cd3b9063288
    url: https://socialweb.coop/activitypub/test-cases/018c3e17-a1bd-7040-8007-4cd3b9063288
    id: urn:uuid:018c3e17-a1bd-7040-8007-4cd3b9063288
  - slug: outbox-post-servers-must-return-a-201-created-http-code
    uuid: 723afcbb-118d-433e-8ab4-560ffca93582
    url: https://socialweb.coop/activitypub/test-cases/723afcbb-118d-433e-8ab4-560ffca93582
    id: urn:uuid:723afcbb-118d-433e-8ab4-560ffca93582
"@context":
  - https://www.w3.org/ns/activitystreams

I can inspect a single test

⚡ activitypub-testing get test --slug actor-must-serve-as2-object-to-get

output

I can run an automated test to determine the result for a test subject

⚡ activitypub-testing run test --slug actor-must-serve-as2-object-to-get --input.time=T10S --input.id='https://socialweb.coop' | jq .
{
  "type": "Assertion",
  "test": {
    "id": "urn:uuid:e7ee491d-88d7-4e67-80c8-f74781bb247c",
    "uuid": "e7ee491d-88d7-4e67-80c8-f74781bb247c",
    "url": "https://socialweb.coop/activitypub/test-cases/actor-must-serve-as2-object-to-get/",
    "slug": "actor-must-serve-as2-object-to-get"
  },
  "input": {
    "time": "T10S",
    "id": "https://socialweb.coop"
  },
  "result": {
    "outcome": "passed"
  },
  "@context": [
    "https://socialweb.coop/ns/testing/context.json",
    "https://www.w3.org/ns/activitystreams"
  ]
}

I can run all the automated tests that are appropriate for a provided test subject and see the results

We already saw activitypub-testing test actor <actor.id> up above. Here we'll show it again, but using the --output='application/vc+ld+json' flag to request a Verifiable Credential representation of the assertions from the test runs.

⚡ activitypub-testing test actor https://socialweb.coop --output vc | jq .

show output

{
  "type": [
    "VerifiableCredential"
  ],
  "id": "urn:uuid:9caf78ef-4711-45c4-9cbd-62ae446972b5",
  "credentialSubject": {
    "id": "https://socialweb.coop/",
    "type": [
      "TestSubject",
      "Actor"
    ],
    "assertions": [
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "following-collection-must-be-a-collection",
          "_:uuid": "018c3e17-a1bd-7040-8007-4cd3b9063288",
          "dct:description": "tests whether an ActivityPub Object has a `following`` collection with an appropriate Collection type",
          "as:name": "An ActivityPub Actor Object's `following` Collection Must be a Collection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/following-collection-must-be-a-collection"
          }
        },
        "rule:input": {
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "outbox-post-servers-must-return-a-201-created-http-code",
          "_:uuid": "723afcbb-118d-433e-8ab4-560ffca93582",
          "dct:description": "This test checks that an ActivityPub Outbox responds with a 201 status code when sent a POST request with an Activity submission.",
          "as:name": "Outbox Servers handling activity submissions MUST return a 201 created HTTP status code",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/outbox-post-servers-must-return-a-201-created-http-code"
          }
        },
        "rule:input": {
          "_:time": "T10S",
          "as:outbox": {
            "@id": "https://socialweb.coop/outbox"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "outbox-must-be-an-orderedcollection",
          "_:uuid": "4af549f4-3797-4d99-a151-67c3d8feaa46",
          "dct:description": "This rule checks whether the outbox property of an object appears to be an OrderedCollection",
          "as:name": "outbox must be an OrderedCollection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/outbox-must-be-an-orderedcollection"
          }
        },
        "rule:input": {
          "_:time": "T10S",
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "@type": "earl:TestResult",
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "inbox-must-be-an-orderedcollection",
          "_:uuid": "5e94d155-ed4a-4d71-b797-d7c387736ecf",
          "dct:description": "This rule checks whether the inbox property of an object appears to be an OrderedCollection",
          "as:name": "inbox must be an OrderedCollection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/inbox-must-be-an-orderedcollection"
          }
        },
        "rule:input": {
          "_:time": "T10S",
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "@type": "earl:TestResult",
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "actor-must-serve-as2-object-to-get",
          "_:uuid": "e7ee491d-88d7-4e67-80c8-f74781bb247c",
          "dct:description": "This rule checks that URLs of ActivityPub objects can be resolved to a representation with well-known media type for further processing.",
          "as:name": "ActivityPub Servers Must Serve Objects in Response to an HTTP GET Request Accepting ActivityStreams 2.0 Media Type",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/actor-must-serve-as2-object-to-get"
          }
        },
        "rule:input": {
          "@id": "https://socialweb.coop/",
          "_:time": "T10S"
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "followers-collection-must-be-a-collection",
          "_:uuid": "018c3e08-611f-7e56-9f45-2fe5e4877d4e",
          "dct:description": "tests whether an ActivityPub Object has a `followers`` collection with an appropriate Collection type",
          "as:name": "An ActivityPub Actor Object's `followers` Collection Must be a Collection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/followers-collection-must-be-a-collection"
          }
        },
        "rule:input": {
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "liked-collection-must-be-a-collection",
          "_:uuid": "018c3df2-d6d8-7f62-805b-b71a96cc6170",
          "dct:description": "tests whether an ActivityPub Object has a liked collection with an appropriate Collection type",
          "as:name": "An ActivityPub Object `liked` Collection Must be a Collection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/liked-collection-must-be-a-collection"
          }
        },
        "rule:input": {
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "likes-collection-must-be-a-collection",
          "_:uuid": "200b9bc8-aae3-46f2-a6ab-5366042c0f6e",
          "dct:description": "tests whether an ActivityPub Object has a likes collection with an appropriate Collection type",
          "as:name": "An ActivityPub Object `likes` Collection Must be a Collection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/likes-collection-must-be-a-collection"
          }
        },
        "rule:input": {
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "shares-collection-must-be-a-collection",
          "_:uuid": "b03a5245-1072-426d-91b3-a3d412d45ae8",
          "dct:description": "This test case tests that an actor's shares collection is an expected kind of collection",
          "as:name": "shares collection MUST be either an OrderedCollection or a Collection",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/shares-collection-must-be-a-collection"
          }
        },
        "rule:input": {
          "as:object": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          }
        }
      },
      {
        "@context": {
          "as": "https://www.w3.org/ns/activitystreams#",
          "earl": "http://www.w3.org/ns/earl#",
          "rule": "https://www.w3.org/TR/act-rules-format/#",
          "dct": "http://purl.org/dc/terms/",
          "foaf": "http://xmlns.com/foaf/spec/",
          "xsd": "http://www.w3.org/2001/XMLSchema#",
          "ldp": "http://www.w3.org/ns/ldp#",
          "vcard": "http://www.w3.org/2006/vcard/ns#"
        },
        "@type": "earl:Assertion",
        "earl:test": {
          "_:slug": "actor-objects-must-have-inbox-outbox-properties",
          "_:uuid": "acaacb5f-8f7e-4f28-8d81-c7955070a767",
          "dct:description": "This rule checks whether a given Actor Object has the properties inbox and outbox.",
          "as:name": "Actor Objects Must Have Properties `inbox` and `outbox`",
          "as:url": {
            "@id": "https://socialweb.coop/activitypub/test-cases/actor-objects-must-have-inbox-outbox-properties"
          }
        },
        "rule:input": {
          "as:actor": {
            "@id": "{\n  \"type\": [\n    \"Organization\"\n  ],\n  \"inbox\": \"https://socialweb.coop/inbox\",\n  \"outbox\": \"https://socialweb.coop/outbox\",\n  \"followers\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"following\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"liked\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"likes\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"shares\": {\n    \"type\": \"OrderedCollection\"\n  },\n  \"id\": \"https://socialweb.coop/\",\n  \"@context\": [\n    \"https://www.w3.org/ns/activitystreams\"\n  ]\n}"
          }
        },
        "as:result": {
          "earl:outcome": {
            "@type": "earl:OutcomeValue",
            "@value": "passed"
          },
          "as:source": []
        }
      }
    ]
  },
  "issuer": "did:web:activitypub-testing.socialweb.coop",
  "validFrom": "2024-01-02T08:26:16.603Z",
  "@context": [
    "https://www.w3.org/ns/credentials/v2"
  ]
}

I can add a test case

You can add a test case by publishing to the web, and using the ones in activitypub-testing as a template. We need all the tests we can get to make sense of the social web. Don't let anyone tell you you have to do it a certain way or in any one place.

If you don't have anywhere else to publish a test case, you can do so by creating a pull request into activitypub-tests in the activitypub-testing repository.

I can give examples of applying my test case

An ActivityPub Test Case is a specification for how to test a test subject. Because it is a specification in its own right, it too can have Test Cases for the Test Specification 🙃 i.e. examples of concrete Input to the test along with test targets derived from the input and the test outcome for each test target. Here is an example.

I can solicit feedback on my test case so I can make it better

Publish your test case on the web, then Announce it to whoever you want via ActivityPub.

If you want feedback from the maintainers of activitypub-testing, make a Pull Request into activitypub-tests in the activitypub-testing repository with a draft of your test case, and we'll provide feedback so you can make it better.

I can discover ideas for tests that need to be newly authored or improved

Browse the activitypub-testing/website /conformance/requirements page to find requirements, then write a test case for any of them you want to.

Tests don't have to just be for Conformance Requirements, though that's where we're starting in order to ensure activitypub-testing can be a Conformance Test. But you are also encouraged to write tests for other ActivityPub behaviors that are merely recommended or optional.

And you should definitely discover ideas for what to test from the Fediverse Enhancement Proposals in the SocialCG Forum for even more protocol behaviors that can and should be tested by activitypub-testing and other ActivityPub test runners.

I can run several testing tools and merge the results

Test results are expressed as Evaluation and Report Language (EARL) Assertions as recommended for Accessibility Conformance Testing by W3C Accessibility Guidelines Working Group in Expressing ACT Rule results with JSON-LD and EARL and used by other W3C groups like W3C Solid CG's Solid QA.

Because the Assertions are serialized as JSON-LD, you can combine them with other JSON-LD Assertions by putting them in a JSON Array. Unlike plain JSON or other non-RDF syntaxes, you can combine these assertions with with Assertions in other non-JSON RDF Syntaxes like xml, HDT, cbor, yaml, or rdfc-1.0.

An example of activtiypub-testing test actor https://socialweb.coop --output=vc as a canonicalized RDF dataset

<https://socialweb.coop/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/ns/credentials/issuer-dependent#Actor> .
<https://socialweb.coop/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/ns/credentials/issuer-dependent#TestSubject> .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n13 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n16 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n18 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n21 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n24 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n27 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n29 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n32 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n35 .
<https://socialweb.coop/> <https://www.w3.org/ns/credentials/issuer-dependent#assertions> _:c14n9 .
<urn:uuid:5c84269d-ff87-4fc8-8080-bc8e8e32bf37> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2018/credentials#VerifiableCredential> .
<urn:uuid:5c84269d-ff87-4fc8-8080-bc8e8e32bf37> <https://www.w3.org/2018/credentials#credentialSubject> <https://socialweb.coop/> .
<urn:uuid:5c84269d-ff87-4fc8-8080-bc8e8e32bf37> <https://www.w3.org/2018/credentials#issuer> <did:web:activitypub-testing.socialweb.coop> .
<urn:uuid:5c84269d-ff87-4fc8-8080-bc8e8e32bf37> <https://www.w3.org/2018/credentials#validFrom> "2024-01-02T09:01:59.084Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
_:c14n0 <http://purl.org/dc/terms/description> "This test checks that an ActivityPub Outbox responds with a 201 status code when sent a POST request with an Activity submission." .
_:c14n0 <https://www.w3.org/ns/activitystreams#name> "Outbox Servers handling activity submissions MUST return a 201 created HTTP status code" .
_:c14n0 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/outbox-post-servers-must-return-a-201-created-http-code> .
_:c14n1 <http://purl.org/dc/terms/description> "This test case tests that an actor's shares collection is an expected kind of collection" .
_:c14n1 <https://www.w3.org/ns/activitystreams#name> "shares collection MUST be either an OrderedCollection or a Collection" .
_:c14n1 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/shares-collection-must-be-a-collection> .
_:c14n10 <http://purl.org/dc/terms/description> "This rule checks that URLs of ActivityPub objects can be resolved to a representation with well-known media type for further processing." .
_:c14n10 <https://www.w3.org/ns/activitystreams#name> "ActivityPub Servers Must Serve Objects in Response to an HTTP GET Request Accepting ActivityStreams 2.0 Media Type" .
_:c14n10 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/actor-must-serve-as2-object-to-get> .
_:c14n11 <http://purl.org/dc/terms/description> "This rule checks whether the inbox property of an object appears to be an OrderedCollection" .
_:c14n11 <https://www.w3.org/ns/activitystreams#name> "inbox must be an OrderedCollection" .
_:c14n11 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/inbox-must-be-an-orderedcollection> .
_:c14n12 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#TestResult> .
_:c14n12 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n13 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n13 <http://www.w3.org/ns/earl#test> _:c14n4 .
_:c14n13 <https://www.w3.org/TR/act-rules-format/#input> _:c14n14 .
_:c14n13 <https://www.w3.org/ns/activitystreams#result> _:c14n12 .
_:c14n15 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#TestResult> .
_:c14n15 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n16 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n16 <http://www.w3.org/ns/earl#test> _:c14n11 .
_:c14n16 <https://www.w3.org/TR/act-rules-format/#input> _:c14n17 .
_:c14n16 <https://www.w3.org/ns/activitystreams#result> _:c14n15 .
_:c14n18 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n18 <http://www.w3.org/ns/earl#test> _:c14n3 .
_:c14n18 <https://www.w3.org/TR/act-rules-format/#input> _:c14n19 .
_:c14n18 <https://www.w3.org/ns/activitystreams#result> _:c14n20 .
_:c14n2 <http://purl.org/dc/terms/description> "This rule checks whether a given Actor Object has the properties inbox and outbox." .
_:c14n2 <https://www.w3.org/ns/activitystreams#name> "Actor Objects Must Have Properties `inbox` and `outbox`" .
_:c14n2 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/actor-objects-must-have-inbox-outbox-properties> .
_:c14n20 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n21 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n21 <http://www.w3.org/ns/earl#test> _:c14n1 .
_:c14n21 <https://www.w3.org/TR/act-rules-format/#input> _:c14n22 .
_:c14n21 <https://www.w3.org/ns/activitystreams#result> _:c14n23 .
_:c14n23 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n24 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n24 <http://www.w3.org/ns/earl#test> _:c14n7 .
_:c14n24 <https://www.w3.org/TR/act-rules-format/#input> _:c14n25 .
_:c14n24 <https://www.w3.org/ns/activitystreams#result> _:c14n26 .
_:c14n26 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n27 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n27 <http://www.w3.org/ns/earl#test> _:c14n0 .
_:c14n27 <https://www.w3.org/TR/act-rules-format/#input> _:c14n6 .
_:c14n27 <https://www.w3.org/ns/activitystreams#result> _:c14n28 .
_:c14n28 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n29 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n29 <http://www.w3.org/ns/earl#test> _:c14n5 .
_:c14n29 <https://www.w3.org/TR/act-rules-format/#input> _:c14n30 .
_:c14n29 <https://www.w3.org/ns/activitystreams#result> _:c14n31 .
_:c14n3 <http://purl.org/dc/terms/description> "tests whether an ActivityPub Object has a likes collection with an appropriate Collection type" .
_:c14n3 <https://www.w3.org/ns/activitystreams#name> "An ActivityPub Object `likes` Collection Must be a Collection" .
_:c14n3 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/likes-collection-must-be-a-collection> .
_:c14n31 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n32 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n32 <http://www.w3.org/ns/earl#test> _:c14n8 .
_:c14n32 <https://www.w3.org/TR/act-rules-format/#input> _:c14n33 .
_:c14n32 <https://www.w3.org/ns/activitystreams#result> _:c14n34 .
_:c14n34 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n35 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n35 <http://www.w3.org/ns/earl#test> _:c14n2 .
_:c14n35 <https://www.w3.org/TR/act-rules-format/#input> _:c14n36 .
_:c14n35 <https://www.w3.org/ns/activitystreams#result> _:c14n37 .
_:c14n37 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n38 <http://www.w3.org/ns/earl#outcome> "passed"^^<http://www.w3.org/ns/earl#OutcomeValue> .
_:c14n4 <http://purl.org/dc/terms/description> "This rule checks whether the outbox property of an object appears to be an OrderedCollection" .
_:c14n4 <https://www.w3.org/ns/activitystreams#name> "outbox must be an OrderedCollection" .
_:c14n4 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/outbox-must-be-an-orderedcollection> .
_:c14n5 <http://purl.org/dc/terms/description> "tests whether an ActivityPub Object has a liked collection with an appropriate Collection type" .
_:c14n5 <https://www.w3.org/ns/activitystreams#name> "An ActivityPub Object `liked` Collection Must be a Collection" .
_:c14n5 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/liked-collection-must-be-a-collection> .
_:c14n6 <https://www.w3.org/ns/activitystreams#outbox> <https://socialweb.coop/outbox> .
_:c14n7 <http://purl.org/dc/terms/description> "tests whether an ActivityPub Object has a `followers`` collection with an appropriate Collection type" .
_:c14n7 <https://www.w3.org/ns/activitystreams#name> "An ActivityPub Actor Object's `followers` Collection Must be a Collection" .
_:c14n7 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/followers-collection-must-be-a-collection> .
_:c14n8 <http://purl.org/dc/terms/description> "tests whether an ActivityPub Object has a `following`` collection with an appropriate Collection type" .
_:c14n8 <https://www.w3.org/ns/activitystreams#name> "An ActivityPub Actor Object's `following` Collection Must be a Collection" .
_:c14n8 <https://www.w3.org/ns/activitystreams#url> <https://socialweb.coop/activitypub/test-cases/following-collection-must-be-a-collection> .
_:c14n9 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/earl#Assertion> .
_:c14n9 <http://www.w3.org/ns/earl#test> _:c14n10 .
_:c14n9 <https://www.w3.org/TR/act-rules-format/#input> <https://socialweb.coop/> .
_:c14n9 <https://www.w3.org/ns/activitystreams#result> _:c14n38 .

I can try out many testing tools and am never locked into just one

If you don't like your test runner, you should be able to run the same conformance tests with a different runner and get similar results. Having one test suite is not obviously better than having zero test suites, because any single test runner or test suite is a single point of failure, lacks accountability without competition, and lacks legitimacy without voluntary adoption

While activitypub-testing currently contains both test cases and a test runner, we believe that what's more important than any one test runner is an open ecosystem of test runners that implement the same test case specifications with consistency such that results from separate runners can be compared and there is fair competition amongst testing tool providers to provide the best tools to the developers and social web actors who want to get tested.

Now what do we do?

We'll keep working on improving activitypub-testing and especially adding test cases until there are tests for every Conformance Reuqirement in the dataset.

File an issue if you have one. I know I do. There is so much to be done in working toward ubiquitous Conformance tests for the ActivityPub network and the rest of the Fediverse.

#GetTested!

Invite your friends to #GetTested. tag it with #ActivityPubTesting.

See more at /activitypub/projects/activitypub-testing/.

― bengo

🌳

Acknowledgements

Tara and Sovereign Tech Fund - for investing in this work (and others!) and for being among the earliest supporters of the activitypub-testing project.
bumblefudge - Leading the W3C SocialCG Testing Task Force since 2023-09-13 and true fren. wagmi.
codenamedmitri - SocialCG Chair and, more importantly, *the* paragon of a consummate web builder. Dmitri's work on solid, DID, VC, VC4EDU, DCC and way more has inspired me for more than half a decade, and he's been an extremely generous technical mentor.
Christine Lemmer-Webber - Has been an effective editor of ActivityPub, one of the few other people to actually make a test suite and gift it, co-organizer of ActivityPub Conf, artist, podcaster, nonprofit computing [institute](https://spritely.institute/) founder/CTO, supportive partner, free culture activist. Truly one of the most inspiring and capable Programmer and Activist I've ever heard of. Just knowing of Christine and following her work has been like a PhD in Computer Science and several other fields.
rhiaro - all the above qualities. Really inspiring human who's advanced the most important parts of the social web at W3C tirelessly for a decade. None of this could have happened without them. I always tell my W3C AC rep to vote for Dr. Amy Guy for TAG and I happily endorse them for any role on any team. rhiaro is a true visionary of a mindful social web.