SPS Commerce REST API Standards Now Open Source

SPS Commerce REST API Standards Now Open Source

The SPS Commerce Tech Team has been working tirelessly over the past few years in curating a single set of technology best practices that we call Guardrails. These Guardrails are the culmination of years of experience from our engineering teams in designing, building, implementing, deploying, and operating best-of-breed software on the world’s largest retail network. These Guardrails consist of high-level technology decisions such as Language and Framework selection or DevOps Practices and drive all the way into expectations around Secret Management and Environments. As you would imagine, it’s not easy aggregating the knowledge and water-cooler discussions from hundreds of engineers spread across multiple continents. The most important aspect of that is achieving singular alignment and agreement of approaches that can be codified into SPS’ internal engineering platform we call Atlas, which is where SPS Commerce is building its next-generation API Platform. On top of both of these initiatives has been the foundational requirement to pursue and dive head-first into API-First culture and align SPS Product, Technology, Processes, and People with clear API Standards and Guidelines for approaching REST-style APIs. SPS Commerce’s first iteration of REST API Standards (also known as a design guide or style guide) is now available publicly and open source.

GitHub Repositoryhttps://github.com/spscommerce/sps-api-standards

Documentationhttps://spscommerce.github.io/sps-api-standards

Why Do We Need API Standards?

REST is an architectural style, and doesn’t provide the clarity and the level of detail necessary to build a set of consistent REST-style APIs without the guidance provided by a style guide. On average 90% of developers use APIs (Stoplight: The API Roadmap, 2022) and spend typically 30% of their time building APIs (Stoplight: The API Roadmap, 2022). That is a massive amount of time and developer experience that can be harnessed and focused on both qualities of life and the efficiency of our engineers.

Historically, SPS Commerce has a long history and experience in building APIs of all kinds, including REST APIs, with hundreds of internal APIs that power the retail network. However, each API, depending on when it was built, which team, which department, and which year, all look substantially different from one another. There is no requirement that these APIs each use different casing for serialization or have different long-running request behaviors or all produce completely different error schemas. In reality, this lack of standardization typically stems from fast-growing organizations without strong governance which can take its toll over time. Spending time upfront to develop API Standards is very time-consuming, but considering the law of diminishing returns, it will pay you back and then some in the long run for both the API Producer and Consumer.

API Producer

  • Make engineering efforts more efficient by enabling assumptions about producing and consuming SPS APIs.
  • Prevent engineers from defining their own standards, allowing them to focus less on undifferentiating engineering efforts.
  • Improve the sustainability and maintainability of an API with clearly identified API change strategies.
  • Guidance is provided for varying use-cases depending on the maturity of an API that enables incremental planned value.

API Consumer

  • Provide a singular product experience across APIs both internally and externally.
  • Lowering the cognitive overhead of APIs which are consistent leads to simpler integrations.
  • Prevent the creation of non-industry standard requests and responses that may cause confusion or may be perceived incorrectly in the community.
  • Provide consistency of standards for simplified publication of documentation.
  • Enabler for standardized SDKs / Libraries that conform to the schema and reusable models.
  • A similar experience between APIs from across SPS makes for a more pleasant experience which increases goodwill with developers.

As SPS Commerce continues to build out its API Platform of the future, unblocking and addressing any gaps in our software development lifecycle is essential. Knowing that engineering teams overwhelmingly identify the lack of API Standardization as a key API challenge and growth concern (SmartBear: The State of API 2020 Report) we knew it would be essential to provide our teams with this clarity and foundational structure to move forward effectively.

https://smartbear.com/resources/ebooks/the-state-of-api-2020-report/

Why REST?

When we use the acronym “API” your engineering mindset might drive you to consider exactly what is meant. API stands for “Application Programming Interface” and is much more generic without the associations we sometimes attribute to it, such as HTTP. APIs generically exist all over our codebases based on that definition. But almost universally across the engineering community, we have come to recognize a particular style of API over HTTP that is often paired with Service-Oriented or Micro-service architectures. On top of that, within the HTTP landscape of APIs, there are a few popular options, including RPC (Remote Procedure Calls), gRPC (gRPC Remote Procedure Calls), GraphQL (Graph Query Language) and of course REST-style (REpresentational State Transfer) APIs. While gRPC can provide some incredible performance gains and GraphQL can enable highly-efficient and broad requests for niche applications, the industry recognizes that REST-style APIs fundamentally is what we would call table stakes for any platform, with over 82% of organizations using REST with OpenAPI specification (SmartBear: The State of API 2020 Report). Of course, there is by no means a limitation in how many API styles you should use, REST-style APIs provide a great foundation for fleshing out your domain-driven design context, which can be used to build highly-functional GraphQL endpoints on top.

“[REST] is simple to use and understand and requires no heavyweight framework or toolchain to get started. It lends itself well to testing and reduces many operational concerns to the same practices used to manage a website. Architecturally, REST has proven scalability and fits in well with domain modeling” (Martin Fowler – Enterprise Integration Using REST 2013).

The Journey

The current state of our REST API Standards was not a project or content that came together over the course of a single agile sprint or iteration. It took work. Contrary to what you might expect, it is more of a people and process problem than a technology problem – as it often turns out to be. Everyone has an opinion. We would not want it any other way. As valuable employees, we are paid to have an opinion. These opinions are based on the culmination of experiences that led an individual through their development journey. That taken into account, the normalization of these opinions and even their collection and aggregation made visible and transparent for intelligent discussion across the organization is a large effort in and of itself. Not all final decisions about SPS Commerce’s REST API Standards will be considered the most “purest” REST-style implementations. Just like the trade-offs, we make every day as engineers, so is it necessary to walk that balance with the API Standards.

To help guide our opinions, restrict bikeshedding, and drive discussions we focused on the following Principles (some provided by CloudFoundy, some our own):

  • Consistency: Understanding how to interact with one resource informs how to interact with any resource. Don’t surprise your users.
  • Discoverability: API responses guide users without the need for external documentation.
  • Simplicity: Complex user workflows are constructed from smaller, easier-to-understand parts (East of Use).
  • Opinionatedness: There is one clear way to do something.
  • Tolerance: Contracts and consumers are as forgiving as possible without compromising security.
  • Automation: Standards and statements should be structured and considered in such a way as to lean towards defendable standards through automation where it does not compromise another design principle.
  • Experience: Developer experience is more of a focus than architecture. DX trumps other principles as a tie-breaker.

We did not start developing the style guide from scratch. There is no need for that. MicrosoftCiscoPaypalCloudFoundry, and many more were used as great starter templates to begin discussion and transformation. It was essential from the start to pull together a cross-functional team of domain technical leads that had a deep understanding of API contract development and solutions within their products and teams. Adding in some individuals who had a clear passion for API Design, we began regular meeting cadences to form a structure and skeleton of our API Standards. Continuous iteration and validation across other engineers enabled us to move to the draft stage of the standards within 9 months of starting. 

Along the way, existing internal APIs were reviewed, and during the development of some of the standards, we pressure tested aspects of them against real live APIs that were getting built. This was helpful to direct the feasibility of some of the styles being developed. Aspects of our style guide present “goldilocks” scenarios that are simpler and easier to implement where the full feature may not be necessary for an initial release. This was especially true when working with standardization of paging and enabling teams to choose between offset (link) and cursor-based paging (link) based on the requirements of their API. Providing a couple of options based on requirements and API maturity empowers the teams to not get caught on unimportant and non-trivial implementation blockers at earlier times in their development.

We intentionally kept the API Standards brief, where possible, with a simple bullet structure knowing that already at the size it was engineers may not even take the time to read through it. As API Handyman, Arnaud Lauret, points out: “nobody cares!”. To drive and aid in transparency of the API Standards we knew from the start they were going to need to be automated to some degree. Using open source linting tool Spectral, a good portion of our API Standards now run linting rules that evaluate the YAML/JSON Open API specification for possible errors and warnings. Standardizing on a single ubiquitous design language for APIs with OpenAPI was the simplest of all the tasks given that SPS Commerce was already well familiar with API Code First patterns and Swagger V2. Initiatives over a couple of years were spent in developing a simplified Developer Experience around building and deploying with Azure Pipelines YAML templates, which made it possible to almost universally rollout API linting with Spectral to all applications that were already centrally publishing their OpenAPI documentation to our SPS API Reference Portal backed by Stoplight. Now pipelines would indicate warning messages and begin to provide a bit more in-your-face transparency to an API and Engineer who may unknowingly deviate from the API Standards. In many cases, engineers are simply not aware of the deviation and automated notifications drive awareness incredibly well. Engineers also “hate” getting linting warnings, and this highly encourages them to clean up their specs, and their APIs and engage in our API Design Review process.

A process and pattern we are still evolving is the API Design Review. It is intended as a simple process for engineers across the organization to submit their API spec for further design review and discussion. This is a collaborative process that encourages API producers to consider many other aspects of API design they had not before. It is a time to educate teams and to pull them into the larger API strategy and taxonomy within the organization to participate in. Issues are submitted on GitHub and follow-up discussions with the key contributors and subject matter experts push for higher value engagements around reusable models & domains, API workflow (how chatty it is), and ultimately aligning the goals with the broader strategy.

Why Open Source?

You might be wondering where the value lay in open sourcing a project like this. While this has begun serving our internal API teams well in producing consistent API specifications, would a team outside of SPS Commerce find this interesting? Short answer: yes we believe so! The SPS Commerce REST API Standards are developed to be almost entirely agnostic of SPS-specific rulesets where possible. This enables you to run our automated spectral rulesets directly from versioned tags and releases on GitHub in your automation if you desire. 

npm install -g @stoplight/spectral-cli

spectral lint openapi.yml --ruleset https://raw.githubusercontent.com/SPSCommerce/sps-api-standards/main/sps-api-standards.spectral.yml

You can also easily inherit and augment our rulesets and add your own:

extends:
  # extend the SPS API Standards and others in composition if desired
  - https://raw.githubusercontent.com/SPSCommerce/sps-api-standards/main/sps-api-standards.spectral.yml
 
rules:
  # disable or downgrade SPS errors to warnings if you desire.
  paths-kebab-case: warn

The high-fidelity documentation provided with Jekyll and GitHub Pages provides clear examples (both correct and incorrect example versions) which connect directly to provided and readily available spectral rulesets that are well tested and verified. Sure, some existing projects contain documentation and rulesets, but are often limited in their scope and lack the continued transparency and commitment to evolving over time. You can keep track of the SPS Commerce REST API Standards as they evolve using GitHub Releases with semantic versions (semver.org) and GitHub Discussions, which should all provide transparency to incoming changes and also act as a decision record.

GitHub Repositoryhttps://github.com/spscommerce/sps-api-standards

Documentationhttps://spscommerce.github.io/sps-api-standards

The intent of making this repository open source is also to enable external visibility of SPS’ Involvement in the external API community, to give back, and to engage with those engineers building on SPS’ APIs externally as needed.

This will be SPS Commerce’s third official Open Source project. You can find out more about our other projects, which include nap and carve on SPS Commerce’s Github Page.

What’s Next?

As mentioned earlier, we are not done. This is just the start. We have a lot of work to do, including continued investment in evolving and adding patterns within the REST API Standards like long-running, batch, and asynchronous requests. Additional work to add more complete coverage on Spectral rulesets for the API Standards will also come in the future. We will be continuing our internal working groups to identify, evaluate and document new standards, patterns and rulesets. We look forward to continuing to engage with the API Community.

Leave a Reply