Which ActivityPub Server Requirements are Easy to Test?

by bengo aka @bengo@mastodon.social

Testing ActivityPub is a daunting task. There are dozens of implementations that don't always work together. The spec has three Conformance Specification Profiles and 50+ requirements refering to 3+ other specifications. Many requirements lack clear testable assertions or otherwise seem complicated to test. It's not always clear where to even begin.

Proposal

Focus on the 'easy to test' ActivityPub server requirements and improve their testability.

Rationale

• get quick wins with low hanging fruit while experimenting with methodological techniques
• if we don't have a methodology that successfully tests easy stuff, web probably won't be able to test hard stuff
• even if we can't agree on which requirements are 'easy to test', the disagreement will lead to better shared understanding of the requirements
• we can improve the testability of a few simple requirements faster than we can improve the testability of all of the requirements including the not-so-simple ones.

Scope

• Only requirements in the ActivityPub Specification and not other documents it refers to
• Only requirements as indicated by RFC2119 "MUST"
• Only requirements relevant to ActivityPub Servers
  • because
    • I want to primarily enable conformance testing of ActivityPub Servers before testing ActivityPub Clients

Determining Difficulty of Server Testing Based on ActivityPub Server Resources

In this paper, we categorize the "Difficulty of Server Testing" of an ActivityPub Server Requirement based on the diversity and cardinality of ActivityPub Server Resources that are used in the requirement and/or a complete verification of the requirement. Generally these resources are either served by the ActivityPub Server test target or managed by the testing environment.

By ActivityPub Server Resources, I mean the various resources that an ActivityPub Server might serve. Often a single server hosts all these resources for an Actor, but it is also possible to consider servers that serve only one or a few of them, and there are some requirements that only describe some of these resources in order to test for conformance.

For example, a requirement that tests ActivityPub Server to Server Interactions may involve an ActivityPub Outbox served by the test target as well as an ActivityPub Actor and ActivityPub Inbox managed by the test environment.

ActivityPub Server Resources described by ActivityPub Requirements

These ActivityPub Server Resources are all parts of an ActivityPub Server:

• ActivityPub Object - generic resource describing any/all objects served by an ActivityPub server, e.g. an Actor, the Actor's outbox, an Activity in the outbox, or another object hosted by the server e.g. one created by an Activity
  • see ActivityPub 3.2. Retrieving objects
• ActivityPub Actor
  • see ActivityPub 4. Actors
• ActivityPub Inbox
• ActivityPub Outbox
• ActivityPub Followers Collection
• ActivityPub Liked Collection
• ActivityPub Likes Collection
• ActivityPub Shares Collection

There may be more resources worth describing, but these are the examples required for this illustration.

Difficulty of Server Testing Categorization

In order to bucket requirements in a way that, however crude, enables focusing on the 'easy to test' ones, we will use the following categories for "Ease of Server Testing" along with criteria sufficient to add a requirement to that category.

• Easy
  • testing requires only ActivityPub Server Resources at test target and none managed by the Test Environment, and can be meaningfully tested by a test client with no special Authorization
• Medium
  • testing requires only ActivityPub Server Resources at the test target and may also require Authorization to POST/GET to these Server Resources, e.g. authorization to post to an Outbox
• Hard
  • testing requires ActivityPub Server Resources other than those at the test target server, e.g. an actor managed by the Test Environment
• NotApplicable
  • the requirement is not relevant to ActivityPub server testing, e.g. it is a requirement that is applicable to an ActivityPub Client but not to a server

Requirements Categorized by Difficulty of Server Testing

Easy to Test ActivityPub Requirements

ActivityPub Object

• 08549639-2888-4ee2-a320-97fc7ee32e00 - when serving objects, a server MUST present the ActivityStreams object in response to request of canonical ActivityPub media type

ActivityPub Actor

• inbox - cc3f730a-37a9-4af9-948f-7c8a0b7f6c41 - actor objects must have a valid inbox property
• outbox - b8647b47-defb-483c-b468-8602d1124169 - actor object must have a valid outbox property

ActivityPub Outbox

• 003a3be2-fb58-4812-a3a3-795067254327 - The outbox MUST be an OrderedCollection.
• 3b925cdf-89fe-4f51-b41f-26df23f58e0c - outbox servers must return a 201 Created HTTP code after accepting an activity at an outbox

ActivityPub Inbox

• 4edf6768-c751-448f-96ac-4ef44cb4291f - The inbox MUST be an OrderedCollection.

ActivityPub Actor Collections

• followers - abef3a0a-d3c4-4dee-a320-b28837d0bcd8 - The followers collection must be either an OrderedCollection or a Collection
• following - a4876ff4-7751-4bc6-91e0-9275382d4a85 - The following collection must be either an OrderedCollection or a Collection
• liked - d2db8da3-25d4-4dd9-9c9c-b2793fd899cf - when actor has a liked collection, it MUST be Collection or OrderedCollection

ActivityPub Object Collections

• likes - f965e989-4084-4f9d-9119-6a7ea13bcb64 - object's likes collection must be either an OrderedCollection or a Collection
• shares - 937ae4e2-dd33-40c7-be1d-3ecac7f9fad5 - The shares collection MUST be either an OrderedCollection or a Collection

Medium Difficulty-of-testing ActivityPub Requirements

This will be added to over time to record categorization decisions, but has been omitted for now in order to focus on requirements that are easy to test.

Hard Difficulty-of-testing ActivityPub Requirements

This will be added to over time to record categorization decisions, but has been omitted for now in order to focus on requirements that are easy to test.

Conclusion

In this post, I've proposed:

Focus on the 'easy to test' ActivityPub server requirements and improve their testability.

I then proposed a method of categorizing the "Difficulty of Server Testing" for ActivityPub requirements by considering the ActivityPub Server Resources described by the requirement and needed in order to test the requirement in a Test Environment.

I applied this method and the result is a list of 11 requirements with Easy Difficulty of Server Testing (out of 50 total ActivityPub Requirements).

Some next steps for me are:

• explain this proposal at w3.org SocialCG Testing TF meeting 2023-10-18 and get feedback
• elaborate on how to improve testability of the easy to test requirements

Some calls to action for you:

• reply to this post using ActivityPub
• review the 11 "easy to test" requirements and
  • consider whether you agree they are easy to test based on your way of categorizing, whether you agree they are easy to test based on the described way of categorizing
  • write down how you would improve the testability of the requirement
  • write down how you would test the requirement
  • self-certify your ActivityPub URL against the requirements