Knowing where all of the APIs you operate and depend on as an enterprise organnization is one of the biggest challenges organizations face while trying to do business online at scale. A problem that is only going to increase when it comes to the growth in the number of enterprise organizations investing in microservices, containers, and continous delivery and deployment (CI/CD).
Stremdata.io Discover is a tool that was born out of over five years of API discovery research, looking to provide a simple, API-driven approach to discovering what web services and APIs exist across an organization, discovering what teams are working on, and discovering what an enterprise's capabilities are when it comes to competing in the digital economy, using web, mobile, device, and network applications.
Finding data, web services, and APIs that already exist across a public website.
Looking for APIs within GitHub organizations within public and private repositories.
Using existing open source solutions to parse log files looking for APIs.
Using AWS Cloudwatch to discover which APIs are being used in operations.
Many enterprise organizations have web service and API directories and catalogs, however they often suffer from being out of date, and lacking the meta data needed to be useful. API discovery will always have a manual component to it, but it is something that will need to be automated if it is going to keep pace with the change of pace that occurs within entenrprise development and IT groups working to meet the demand put on them.
To help automate the discovery of APIs that are already in use across the enterprise, teams need to be taking advantage of the data outputed by existing components of the API stack, leveraging automation to sniff out API signals that exist within common logs. Web server, database, DNS, and other essential building blocks often produce the log files needed to automate the discovery of APIs that are driving web and mobile applications.
Extracting API signals from log files, .Har files, and other common outputs.
Ingesting WSDL documents for web services available across the platform.
Ingesting OpenAPI 2.0 or 3.0 specifications into the catalog.
Ingesting AsyncAPI 1.0+ specifications into the catalog.
Ingesting JSON Schema into the API discovery catalog.
New items can be queued up and executed on a regular schedule.
Allows for addition of API definitions and discovery files via email.
Post APIs.json and OpenAPI files via the Discover API.
To be able to handle the amount of data being gathered by Streamdata.io Discover automated agents, a structured, machine readable approach is needed to handle the intake of API definitions being identified. Discover works with each agent to organize API signals that are gathered into OpenAPI definitions, wrapped as intake APIs.json collections, documenting what has been done, and managing intake in an orderly fashion.
Streamdata.io Discover provides a solution to managing the intake of API definitions into the system, aggregating signals that have been discovered, converting them into a common format, and stacking into queues for processing on a regular schedule. Making signals available for archiving, or used to help define API contracts that are already in use, while working to move them forwarding as part of the overall API Journey.
Standardized information to apply across API contracts.
Managing a central pool of servers used to deliver on API contracts.
Ensuring that every capability of an API contract is represented as a path.
Defining all the components that go into delivering on each API contract.
Defining how each service is secured, documenting the IAM layer for the contract.
Using tags to organize all OpenAPI contracts for more meaningful use across the Lifecycle.
API discovery is not just about finding out about the APIs that already exist, and are in use. API discovery is about defining the enterprise capabilities that exist, and working to improve upon them at scale. OpenAPI contracts are how leading API providers are defining the business contract that exists between provider and consumer, providing a framework for the business rules that exist, but done in a machine readable way.
Streamdata.io Discover is all about discovering existing capabilities that exist across the enterprise API landscape, better define them as consistent API contracts, and hopefully move them forward. Combining manual and automated API discovery results to hep define and refine upon API contracts that are being used, helping harden them, and make the more meaningful in a technical, as well as business capacity.
Centrally manage the parameters in use across all APIs in the system.
Centrally manage the headers in use across all APIs in the system.
Centrally manage the request bodies in use across all APIs in the system.
Centrally manage the responses in use across all APIs in the system.
Centrally manage the examples in use across all APIs in the system.
Centrally manage the security schemes in use across all APIs in the system.
Centrally manage the schema in use across all APIs in the system.
Centrally manage the links in use across all APIs in the system.
Centrally manage the callbacks in use across all APIs in the system.
Streamdata.io Discover embraces the latest release of the OpenAPI specification, and puts the new components object to work, helping aggregate, organize, and hopefuly evolve the common parts that go into delivering APIs consistently at scale. The core of Discover is based upon the OpenAPI specification, and leverages the ability to use components in defining API signals found, while also helping mature and evolve them as part of API contracts.
The next generation of the OpenAPI specification is essential to helping the enterprise getting a handle on not just what APIs exist across organizations and groups, but also move API operations towards a more strategy approach to API governance. The components portion of Discover helps API teams think more constructively about how they define and design their APIs, but also do it more efficiently, and consistently at scale.
Leveraging OpenAPI to define and make all HTTP 1.1 capabilities discoverable.
Leverage ASyncAPI to define and make all HTTP/2 capabilities discoverable.
Leverage ASyncAPI to define and make all HTTP/3 capabilities discoverable.
Leverage ASyncAPI to define and make all TCP capabilities discoverable.
Leverage ASyncAPI to define and make all MQTT capabilities discoverable.
Leverage ASyncAPI to define and make all AMQP capabilities discoverable.
API discovery isn't just about knowing where our HTTP / Web APIs, and there are numerous protocls in use across the enterprise, making it important that API discover covers multiple approaches to actually delivering APIs that supports many different services, tools, and protocols.
Defining the important topics consumers can subscribe to.
Making the streaming layer of API operations more discoverable.
Defining the most meaningful events that occur across the platform.
Working to make all messages discoverable across API operations.
Discover pushes the boundaries of API discovery to include event-driven infrastructure, going well beyond the request and response infrastructure that has become ubiquitous, and making sure API discovery includes the indexing of the most relevant and meaningful topics, streams, events, messages, and other real time aspects of doing business.
Using collections to manage different types of API collections being used across the enterprise.
Providing the operational level details for each API being made as part of an API collection.
Helping ensure that every API collection has an active owner and maintainer advocating for its existence.
Using tags to define, organize, and make different collections of APIs available across the enterprise.
The OpenAPI specification is the core of Streamdata.io Discover, providing a machine readable standard for defining the surface area of each API contract. It also uses the APIs.json specification to provide a machine readable standard for defining the elements of API operations in place to support each API, ensuring that all APIs have the essential building they need to operate reliabily, and are indexed and discoverable.
Streamdata.io Discover provides a standardized way to quantifying and organizing the APIs that exist across the enterprise, and allows them to be organized into different collections. Providing a portable, machine readable, API-driven approach to building API catalogs, directories, and collections that help us identify, organize, and evolve the digital resources that exist across the enterprise API landscape.
Reference to the machine readable definition for each API contract.
Reference to the executable run-time definition for each API.
Reference to the event-driven definition for each API contract.
Where developers can find interactive documentation for the API, driven by the OpenAPI definition.
Providing the links where consumers can find more information on limitations placed on each API.
Pointing API consumers to where they can sign in or login to get access to an API, and other resources.
Providing a link to the status of an API, with up to date information on the availability of each API.
Providing a reference to the legal terms of service applied to an API and its operations.
Making sure there is a link to the support resources available for each individual API resource.
Sharing details of each services GitHub repository, Gist, or other code resources that can be used in the pipeline.
Sharing details of each services GitLab repository, and other code resources that can be used in the pipeline.
Providing the link to security practices that are being applied to each individual API as part of operations.
Providing the link to service level agreement (SLA) that governs each API, providing the details behind the operation of each service.
Assigning a score to each property type, and tallying the scores when they are applied to individual APIs, allowing APIs to be ranked across operations.
In addition to having a reference to the machine readable contract for each API, it is important to esnure that API collections possess a variety of properties that help ensure that an API is being operated in accordance with a common strategy. Providing references to common building blocks of API operations for API consumers to use when on-boarding with an API, but also using them to understand the quality of services being delivered across teams.
Streamdata.io Discover leverages the APIs.json format to provide a variety of common API properties that can be used to make APIs more discoverable, and usable throughout their lifecycle. Providing human and machine readable properties that can significantly contribute to the on-boarding and integration with API resources, while also using them as a set of metrics for determining the overall maturity of each APIs being delivered.
Manage and put to use Postman Environment files as part of single, or across many APIs.
Manage environments using the GitLab Environments API, and centralizing client access.
We depend on a variety of critical variables to put internal, partner, and 3rd party API resources documented as part of any API discovery effort. One of the most essential properties to determine the maturity, quality, and value an API possesses, will be it's environment settings. Allowing API consumers to actually connect with an API, and begin putting it to use in API clients like Postman, and other integration tooling.
Streamdata.io Discover doesn't reinvent the wheel when it comes to API environment management, putting to use existing approaches to managing the integrations in use across API infrastructure. OpenAPI, Postman, and other common services and tooling are already providing environment management solutions, and APIs, so we decide to just use what already is in use, but provide a way to organize and index as part of API collections.
Providing basic keyword search across all APIs documented with the system.
Serach for APIs based upon the domain, subdomain, or other server details.
Search for APIs based upon the organization that developed or owns the service.
Search for APIs based upon the user that developed or maintains the service.
Search for APIs basedu pon the type of collection that it is part of in the system.
Search for APIs based upon the property types the possess as part of their definition.
Search for APIs based upon their maturity, and having met their service level agreement.
Search for APIs based upon where they are in the lifecycle from design to production.
Knowing where your API resources are, having them organized into collections as machine readable indexes, opens up a whole new world of search, allowing enteprise resources to continually be discovered. Providing a rich source of information about what resources are available, what enterprise groups are building, and way to continous discover what is being built, used, and driving business across distributed organizations.
Streamdata.io Discover provides a search solution that allows for the precise search of OpenAPI collections, going beyond just basic keywords search, and drilling down into the building blocks of API contracts. Making API discovery begin with finding, defining, and evolving the API resources that exist across the enterprise, but then contine to deliver discovery capabilities across the API lifecycle,
Export and publish OpenAPI 2.0 and 3.0 definitions for any API in the catalog.
Export and publish APIs.json definitions for any API collection in the catalog.
Publish an API collection to GitHub as it's own repository within any organization.
Publish JSON autocomplete for paths, headers, parameters, and schema.
Discoverable, machine readable API definitions do not mean much if they cannot be put to work across the API lifecycle. The definition of APIs, and the operations that support them, need to be exportable and published as machine readable definitions that can be used throughout the API lifecycle. Making API contracts available for hardening and maturing as part of API design workflows, then on to development and production via CI/CD pipelines.
Streamdata.io Discover allows for the seamless integration, exporting, and continuous delivery of hardened and mature API contracts, so that they can be applied throughout the lifecycle of an API. Making API discovery not just the place to begin the API journey, but where to discover what is possible throughout the software delivery supply chain--making the API definitions published from Streamdata.io Discover essential to doing business efficiently online.
