Testing Ecosystem Wishlist

User Stories

As a

• end-user
  • I can see test assertions about all the service providers I use so I know which ones may not work as expected
  • I can see test assertions about service providers I am evaluating for future usage so I can choose one that will reliably interoperate and which will maximally provide end-user benefits afforded for by the protocol specification
• test operator e.g. a protocol implementor testing their implementation
  • I can list all the tests that are available to me
  • I can inspect a single test
  • I can run an automated test to determine the result for a test subject
  • I can run all the automated tests that are appropriate for a provided test subject and see the results
  • I can easily explore the test results
  • I can show off my awesome test results
• test author
  • I can add a test case
  • I can give examples of applying my test case
  • I can solicit feedback on my test case so I can make it better
  • I can discover ideas for tests that need to be newly authored or improved
• service provider
  • I can discover the results of tests about the service I am responsible for
  • I can easily explore the test results
  • I can get advice on actions I can take to improve the results of tests run on my system
  • I can run several testing tools and merge the results
  • I can try out many testing tools and am never locked into just one