Streamdata.io Discover

Knowing where all of your digital assets are today.

Knowing where all of the APIs you operate and depend on as an enterprise organnization is one of the biggest challenges organizations face while trying to do business online at scale. A problem that is only going to increase when it comes to the growth in the number of enterprise organizations investing in microservices, containers, and continous delivery and deployment (CI/CD).


Stremdata.io Discover is a tool that was born out of over five years of API discovery research, looking to provide a simple, API-driven approach to discovering what web services and APIs exist across an organization, discovering what teams are working on, and discovering what an enterprise's capabilities are when it comes to competing in the digital economy, using web, mobile, device, and network applications.

Automated Agents

Website

Finding data, web services, and APIs that already exist across a public website.

blog icon

GitHub

Looking for APIs within GitHub organizations within public and private repositories.

Filebeat & Logstash

Using existing open source solutions to parse log files looking for APIs.

AWS Cloudwatch

Using AWS Cloudwatch to discover which APIs are being used in operations.

Agents

API discovery can't keeep up with pace of change.

Many enterprise organizations have web service and API directories and catalogs, however they often suffer from being out of date, and lacking the meta data needed to be useful. API discovery will always have a manual component to it, but it is something that will need to be automated if it is going to keep pace with the change of pace that occurs within entenrprise development and IT groups working to meet the demand put on them.


To help automate the discovery of APIs that are already in use across the enterprise, teams need to be taking advantage of the data outputed by existing components of the API stack, leveraging automation to sniff out API signals that exist within common logs. Web server, database, DNS, and other essential building blocks often produce the log files needed to automate the discovery of APIs that are driving web and mobile applications.

API Intake

Signals

Extracting API signals from log files, .Har files, and other common outputs.

WSDL

Ingesting WSDL documents for web services available across the platform.

OpenAPI

Ingesting OpenAPI 2.0 or 3.0 specifications into the catalog.

AsyncAPI

Ingesting AsyncAPI 1.0+ specifications into the catalog.

JSON Schema

Ingesting JSON Schema into the API discovery catalog.

Queue

New items can be queued up and executed on a regular schedule.

Email

Allows for addition of API definitions and discovery files via email.

API

Post APIs.json and OpenAPI files via the Discover API.

Intake

Ingesting API signals gathered by automated agents.

To be able to handle the amount of data being gathered by Streamdata.io Discover automated agents, a structured, machine readable approach is needed to handle the intake of API definitions being identified. Discover works with each agent to organize API signals that are gathered into OpenAPI definitions, wrapped as intake APIs.json collections, documenting what has been done, and managing intake in an orderly fashion.


Streamdata.io Discover provides a solution to managing the intake of API definitions into the system, aggregating signals that have been discovered, converting them into a common format, and stacking into queues for processing on a regular schedule. Making signals available for archiving, or used to help define API contracts that are already in use, while working to move them forwarding as part of the overall API Journey.

Contracts (OpenAPI 3.0)

Information

Standardized information to apply across API contracts.

blog icon

Servers

Managing a central pool of servers used to deliver on API contracts.

Paths

Ensuring that every capability of an API contract is represented as a path.

Components

Defining all the components that go into delivering on each API contract.

Security

Defining how each service is secured, documenting the IAM layer for the contract.

Tags

Using tags to organize all OpenAPI contracts for more meaningful use across the Lifecycle.

Contracts (OpenAPI 3.0)

Building solid OpenAPI contracts within the enterprise.

API discovery is not just about finding out about the APIs that already exist, and are in use. API discovery is about defining the enterprise capabilities that exist, and working to improve upon them at scale. OpenAPI contracts are how leading API providers are defining the business contract that exists between provider and consumer, providing a framework for the business rules that exist, but done in a machine readable way.


Streamdata.io Discover is all about discovering existing capabilities that exist across the enterprise API landscape, better define them as consistent API contracts, and hopefully move them forward. Combining manual and automated API discovery results to hep define and refine upon API contracts that are being used, helping harden them, and make the more meaningful in a technical, as well as business capacity.

Components (OpenAPI 3.0 & AsyncAPI)

Parameters

Centrally manage the parameters in use across all APIs in the system.

blog icon

Headers

Centrally manage the headers in use across all APIs in the system.

Request Bodies

Centrally manage the request bodies in use across all APIs in the system.

Responses

Centrally manage the responses in use across all APIs in the system.

Examples

Centrally manage the examples in use across all APIs in the system.

Security Schemes

Centrally manage the security schemes in use across all APIs in the system.

Schema

Centrally manage the schema in use across all APIs in the system.

Links

Centrally manage the links in use across all APIs in the system.

Callbacks

Centrally manage the callbacks in use across all APIs in the system.

Components (OpenAPI 3.0 & AsyncAPI)

Making the API lifecycle more modular.

Streamdata.io Discover embraces the latest release of the OpenAPI specification, and puts the new components object to work, helping aggregate, organize, and hopefuly evolve the common parts that go into delivering APIs consistently at scale. The core of Discover is based upon the OpenAPI specification, and leverages the ability to use components in defining API signals found, while also helping mature and evolve them as part of API contracts.


The next generation of the OpenAPI specification is essential to helping the enterprise getting a handle on not just what APIs exist across organizations and groups, but also move API operations towards a more strategy approach to API governance. The components portion of Discover helps API teams think more constructively about how they define and design their APIs, but also do it more efficiently, and consistently at scale.

A Diverse API Toolbox

http

HTTP 1.1

Leveraging OpenAPI to define and make all HTTP 1.1 capabilities discoverable.

HTTP/2

HTTP/2

Leverage ASyncAPI to define and make all HTTP/2 capabilities discoverable.

blog icon

HTTP/3

Leverage ASyncAPI to define and make all HTTP/3 capabilities discoverable.

blog icon

TCP

Leverage ASyncAPI to define and make all TCP capabilities discoverable.

blog icon

MQTT

Leverage ASyncAPI to define and make all MQTT capabilities discoverable.

blog icon

AMQP

Leverage ASyncAPI to define and make all AMQP capabilities discoverable.

A Diverse API Toolbox

The building bocks of scalable API operations.

API discovery isn't just about knowing where our HTTP / Web APIs, and there are numerous protocls in use across the enterprise, making it important that API discover covers multiple approaches to actually delivering APIs that supports many different services, tools, and protocols.

Messaging & Event-Driven (AsyncAPI)

Topics

Topics

Defining the important topics consumers can subscribe to.

Streams

Streams

Making the streaming layer of API operations more discoverable.

Events

Events

Defining the most meaningful events that occur across the platform.

Messages

Messages

Working to make all messages discoverable across API operations.

Messaging & Event-Driven (AsyncAPI)

.

Discover pushes the boundaries of API discovery to include event-driven infrastructure, going well beyond the request and response infrastructure that has become ubiquitous, and making sure API discovery includes the indexing of the most relevant and meaningful topics, streams, events, messages, and other real time aspects of doing business.

Organizing Collections (APIs.json)

Types

Using collections to manage different types of API collections being used across the enterprise.

blog icon

APIs

Providing the operational level details for each API being made as part of an API collection.

Maintainers

Helping ensure that every API collection has an active owner and maintainer advocating for its existence.

Tags

Using tags to define, organize, and make different collections of APIs available across the enterprise.

Collections (APIs.json)

Organizing APIs into meaningful collections.

The OpenAPI specification is the core of Streamdata.io Discover, providing a machine readable standard for defining the surface area of each API contract. It also uses the APIs.json specification to provide a machine readable standard for defining the elements of API operations in place to support each API, ensuring that all APIs have the essential building they need to operate reliabily, and are indexed and discoverable.


Streamdata.io Discover provides a standardized way to quantifying and organizing the APIs that exist across the enterprise, and allows them to be organized into different collections. Providing a portable, machine readable, API-driven approach to building API catalogs, directories, and collections that help us identify, organize, and evolve the digital resources that exist across the enterprise API landscape.

Collection Properties (APIs.json)

OpenAPI

Reference to the machine readable definition for each API contract.

Postman Collections

Reference to the executable run-time definition for each API.

AsyncAPI

Reference to the event-driven definition for each API contract.

Documentation

Where developers can find interactive documentation for the API, driven by the OpenAPI definition.

Plans

Providing the links where consumers can find more information on limitations placed on each API.

Sign Up

Pointing API consumers to where they can sign in or login to get access to an API, and other resources.

Status

Providing a link to the status of an API, with up to date information on the availability of each API.

Terms of Service

Providing a reference to the legal terms of service applied to an API and its operations.

Support

Making sure there is a link to the support resources available for each individual API resource.

GitHub

Sharing details of each services GitHub repository, Gist, or other code resources that can be used in the pipeline.

GitLab

Sharing details of each services GitLab repository, and other code resources that can be used in the pipeline.

Security

Providing the link to security practices that are being applied to each individual API as part of operations.

Service Level Agreement

Providing the link to service level agreement (SLA) that governs each API, providing the details behind the operation of each service.

Rating

Assigning a score to each property type, and tallying the scores when they are applied to individual APIs, allowing APIs to be ranked across operations.

Collection Properties (APIs.json)

The building bocks of scalable API operations.

In addition to having a reference to the machine readable contract for each API, it is important to esnure that API collections possess a variety of properties that help ensure that an API is being operated in accordance with a common strategy. Providing references to common building blocks of API operations for API consumers to use when on-boarding with an API, but also using them to understand the quality of services being delivered across teams.


Streamdata.io Discover leverages the APIs.json format to provide a variety of common API properties that can be used to make APIs more discoverable, and usable throughout their lifecycle. Providing human and machine readable properties that can significantly contribute to the on-boarding and integration with API resources, while also using them as a set of metrics for determining the overall maturity of each APIs being delivered.

Managing Environments

Postman

Manage and put to use Postman Environment files as part of single, or across many APIs.

blog icon

GitLab

Manage environments using the GitLab Environments API, and centralizing client access.

Environments

Managing environments across API consumption.

We depend on a variety of critical variables to put internal, partner, and 3rd party API resources documented as part of any API discovery effort. One of the most essential properties to determine the maturity, quality, and value an API possesses, will be it's environment settings. Allowing API consumers to actually connect with an API, and begin putting it to use in API clients like Postman, and other integration tooling.


Streamdata.io Discover doesn't reinvent the wheel when it comes to API environment management, putting to use existing approaches to managing the integrations in use across API infrastructure. OpenAPI, Postman, and other common services and tooling are already providing environment management solutions, and APIs, so we decide to just use what already is in use, but provide a way to organize and index as part of API collections.

Search

Keyword

Providing basic keyword search across all APIs documented with the system.

blog icon

Server

Serach for APIs based upon the domain, subdomain, or other server details.

Organizations

Search for APIs based upon the organization that developed or owns the service.

Users

Search for APIs based upon the user that developed or maintains the service.

Type

Search for APIs basedu pon the type of collection that it is part of in the system.

Properties

Search for APIs based upon the property types the possess as part of their definition.

Maturity

Search for APIs based upon their maturity, and having met their service level agreement.

Lifecycle

Search for APIs based upon where they are in the lifecycle from design to production.

Search

Knowing where all of your digital assets are across the enterprise.

Knowing where your API resources are, having them organized into collections as machine readable indexes, opens up a whole new world of search, allowing enteprise resources to continually be discovered. Providing a rich source of information about what resources are available, what enterprise groups are building, and way to continous discover what is being built, used, and driving business across distributed organizations.


Streamdata.io Discover provides a search solution that allows for the precise search of OpenAPI collections, going beyond just basic keywords search, and drilling down into the building blocks of API contracts. Making API discovery begin with finding, defining, and evolving the API resources that exist across the enterprise, but then contine to deliver discovery capabilities across the API lifecycle,

Publish

OpenAPI

Export and publish OpenAPI 2.0 and 3.0 definitions for any API in the catalog.

APIs.json

Export and publish APIs.json definitions for any API collection in the catalog.

blog icon

GitHub

Publish an API collection to GitHub as it's own repository within any organization.

Autocomplete

Publish JSON autocomplete for paths, headers, parameters, and schema.

Continuous Publishing

Extend the value of discovery across the lifecycle.

Discoverable, machine readable API definitions do not mean much if they cannot be put to work across the API lifecycle. The definition of APIs, and the operations that support them, need to be exportable and published as machine readable definitions that can be used throughout the API lifecycle. Making API contracts available for hardening and maturing as part of API design workflows, then on to development and production via CI/CD pipelines.


Streamdata.io Discover allows for the seamless integration, exporting, and continuous delivery of hardened and mature API contracts, so that they can be applied throughout the lifecycle of an API. Making API discovery not just the place to begin the API journey, but where to discover what is possible throughout the software delivery supply chain--making the API definitions published from Streamdata.io Discover essential to doing business efficiently online.

Streamdata.io Discover Is Coming Soon!

Contact Us To Learn More