This article describes how you can use the Custom Metrics feature to capture data from APIs. This could be the API of a technical partner you are using for your business, or your own API that you’ve built to support your own customers or processes. It could even be a custom API you’ve built to collect or calculate your internal business metrics or platform metrics. Test tools and monitoring solutions help you to ensure your APIs work correctly, and this should be a vital part of your platform operations. But you can do much more to gain insight in your platform and business health! Custom API data monitoring doesn’t just check uptime and perform data validation - it captures the data produced by your API, and visually displays how platform and business data progresses over time.
Revisiting API monitoring
You can use test tools like Postman, Insomnia, SoapUI or Swagger UI to test your APIs and inspect data for ad-hoc testing and debugging. You can use a synthetic monitoring tools (like Uptrends’ API monitoring) to continuously monitor the availability and response times. And not just that - you’ll want to set up in-depth content validation of your JSON data (using JSON expressions) or XML data (using XPath queries) so you can really check that the data returned by the API meets your expectations - without errors, and within acceptable boundaries. This helps to ensure that the processes depending on those APIs continue to work correctly and swiftly. These processes can range from your mobile apps and customer-facing web apps to business processes in your backend.
Tracking API data
Imagine you have a REST API that is part of an e-commerce system that sells products. The entire product catalog consists of several product lines that come from different suppliers, who update their product list several times a day. You have some background processes that update your own product catalog as soon as updates are available (probably using your suppliers’ API, too). Your Product API is directly used by your e-commerce web shop and your mobile app.
Checking that your product list contains the expected number of products and categories (a certain minimum at least) is rather vital for your business. You probably have some reports about that coming from your backend processes, but monitoring it based on your API data gets you the information much faster.
No problem - you can create a Multi-Step API monitor (MSA) that performs one or more calls to your Product API endpoint that returns JSON-formatted data containing the total number of products, the product categories, and the number of products in each category. You add a few assertions that take the individual attributes (let’s look at TotalProductCount and CategoryCount for now) and verify that those numbers don’t exceed their minimum values. For example, TotalProductCount should always be greater than 1500, and CategoryCount should be at least 10. These assertions are safeguards against any unnoticed product update failures that suddenly empty a large part of your product inventory. If something goes wrong, you’ll be alerted by Uptrends and you can act swiftly. That part of your day-to-day Operations is now covered.
Detecting data trends
But there’s more you can do with this data. Each individual monitor check tells you that your product catalog looks healthy (or not). The product and category count numbers were validated during each check, but you can’t clearly see what the actual values were. In fact, it would be nice to see how the number of products fluctuates during the day (micro changes) and during the month and year (macro changes). If the number of products grows faster than you expected, for example, this could point to a problem in removing retired products from your database, which potentially slows down your API, and your mobile app as a result. Detecting this using simple assertions is hard - you need trend analysis!
You’ve already identified your metric TotalProductCount in the MSA assertion, so it’s very easy to now start collecting its data as custom metric data. Let’s set it up!
How to set up custom API monitoring
First, make sure you have a working MSA monitor setup (which can be just a single step, it doesn’t have to be multi-step) as discussed in the previous section. At least one of the steps in the MSA monitor should define a variable that captures a numeric value from the API response. You can read about creating variables in API monitoring steps to learn more.
Once you have at least one variable in one of your steps, you can scroll down the page to the Custom metrics section. There, you’ll notice that the variables produced by your MSA monitor appear in the list under Variable name.
All you need to do now, is “promote” that variable to a custom metric by giving it a name under Metric name. It can be the same name as the variable that feeds into it, but you may want to use a name that looks good in reporting. For example, if you’re using the variable TotalProductCount, a metric name Total product count would probably work well.
You can add up to 5 custom metrics like this within one MSA monitor. If you need more, please contact Support.
Finally, save your monitor definition so the new custom metric starts to populate!
Inspecting the data and troubleshooting
Before we start setting up charts and tables, let’s inspect our new data first. Navigate to the Monitor Log, locate your MSA monitor and find a recent check result. Click on it to open up the check result popup.
In the Check details section in the popup, notice how the value(s) for your new custom metric(s) appear below the step results. This gives you direct access to the individual custom metric values as they were captured during the execution of the MSA monitor.
If you don’t see the custom metric value you were expecting to see, consider the following: - Did you accidentally open some check results for an older check that was executed when the new custom metric hadn’t been defined yet? - Is your new custom metric capturing numeric integer data? If it contains text data or non-integer numbers (such as 99.9% or 3.1415) then the custom metric won’t be captured. Only integer numbers are supported right now. Please contact us if you want to discuss your needs. - If something went wrong during the execution of the MSA monitor, the variable that feeds into your custom metric may not have been created at all. Our support engineers can work with you to fix those issues.
Creating reports
When you have verified that your new custom metric is producing proper results, it’s time to create a report that will show the progression of your metric, once you’ve collected enough data. For this example, let’s create a new dashboard (more on creating dashboards). Click Dashboards > Add dashboard to create a brand new dashboard. Click on one of the empty tiles, and scroll down in the Add new tile popup to select the tile Custom metrics chart.
The new tile doesn’t display anything at first. Open the tile settings, and locate the monitor that contains the custom metric. Expand it to add the custom metric to your selection. Notice that you can select more than one custom metric: they will be displayed in one chart so you can compare their values.
Similarly, you can use the Custom metrics list tile to create a grid that displays the numeric values for one or more custom metrics. Please note that the displayed values will be average values.
After designing your dashboard, you can save it, export it, and schedule it for automatic delivery via email.
Inspecting raw data
The reports we’ve just looked at will give you a good way to view data trends. But you may want to dig deeper, and work with the underlying data. If you have access to each individual metric value we’ve captured, you can detect sudden spike values more precisely, do min/max/median analysis, and so on.
To do this, create an export file to download all captured metrics to an Excel file. First, navigate to the Monitor Log dashboard to select the appropriate monitor(s). Next, open up the tile settings for the monitor log tile. On the Export tab, make sure to select the Custom metrics checkbox. Click Set to confirm your settings.
Finally, use the dashboard menu (hamburger icon) to start the Excel export process (either directly or through email). Once you have the Excel file, you’ll see separate columns for each custom metric that was captured for your monitor(s).
Custom metric names
When you’re creating a new custom metric, it’s recommended to use a name that’s easy to read in your reporting. The metric’s name, combined with the name of the monitor it’s part of, will appear in the list of available custom metrics when you add it to your custom API data report.
If you are planning to create multiple custom metrics that represent the same kind of data (i.e. the same metric but representing a different group, product, customer, etc.), it makes sense to give them the exact same name. You’ll always be able to tell them apart because of the different monitor names.
When you’re creating custom metrics, it’s good to give it the correct name right away. You can change a metric’s name later on, but keep in mind that it will be treated as a different metric as soon as it’s renamed: the metric is uniquely identified by its name.