Let Splunk Enterprise and ServiceNow really work together

This is part 3 of the “Splunk and CMDB/CSM/IT4IT blog series”. This post is focused on letting ServiceNow CMDB (CSDM) and IPC working together with Splunk Enterprise, Splunk Enterprise Security, and Splunk IT Service Intelligence. It is one of the use cases of my Common/Corporate Metadata Data Management (CMDM) solution by doing the Splunk and ServiceNow integration.

What is there on the Splunk market already?

Well of course there is the “Splunk Add-on for ServiceNow” on Splunk base. That add-on collects incident, event, change, user, user group, location, and CMDB CI information from ServiceNow. And it also provides Splunk alert actions to create ServiceNow events or incidents.

The add-on is doing a great job from a Splunk alert perspective but all of the others are more technical than functional. All the things this add-on is collecting are stored in Splunk event indices. But what about relationships between all of those objects it collects? How do I know the status of a certain CI? How can I find out the servers (hosts) belonging to a certain application of business service? Oh yes, it comes with some means of transforming those time-based collections into lookups (kvstore) but still, I cannot answer the mentioned questions.
Or let me rephrase it more accurately, it is really difficult if not impossible for users of Splunk to do so.

So what is missing then?

Looking at all of those Splunk installations I’ve worked with the last seven years I found:

• There are a lot of shadow CMDBs within Splunk: Tags, lookups, etc. This is a huge problem in the area of who is maintaining them, what about the adoption and quality of the central ServiceNow CMDB/CSDM, how to really correlate Splunk data from server (host) multiple relationships up to the business service, etc.
• Splunk knowledge objects (dashboards, reports, etc) were sometimes hardcoded with underlying infrastructure details. That’s because the application/DevOps teams would like to have dashboards showing the health of their application or service.
• How can Splunk and ServiceNow tooling be used in conjunction together to help the application/DevOps teams to deliver better outcomes?
• With the upcoming IT4IT reference architecture, how to increase the adoption of the service or product model backbone in the organization?
• How not only visualize relationships with Splunk but search on relationships within Splunk.

More details about some of the above points can be found at Splunk – ServiceNow.

Mission

Working with Splunk Enterprise for over 7 years mainly in the IT Operations space but also the security space they all wanted to have a functional integration instead of only technical. And yes I’ve worked with the mentioned add-on at all of my customers but always with limited success.

In my opinion the functional integration should do the following:

• Within Splunk have an overview of all of the (open, new, etc.) Incidents, Changes, and/or Problems. And not searching for it in an index but just right at your fingertips with the latest details.
• Within Splunk can use the CMDB information (content AND CONTEXT) for searching data and power the Splunk knowledge objects.
• Within Splunk can use all of the ServiceNow CSDM objects to power Splunk knowledge objects.
• Being able to list all the (open, new, etc.) Incidents, Changes, and/or Problems belonging to a ServiceNow support group certain users belong to. So not just a flat list of IPC records.
• Asking for the status of a certain CI or one of the IPC records you always get the latest one that matches ServiceNow.

Functional integration

The functional integration between ServiceNow with Splunk Enterprise and Splunk IT Service Intelligence is one of the first use cases for the Common/Corporate Metadata Data Model (CMDM) solution I created.

With the CMDM solution users of Splunk can now reuse ServiceNow CMDB/CSDM content AND CONTEXT directly within Splunk. So removing the need for what I call host-based tags. The following example shows how to get Business Application information out of the ServiceNow CMDB directly within Splunk given a certain IP-address. A use case that security people often need to execute and which till now is mostly done on not frequently updated information or difficult other constructions including using excel.

With the CMDB solution users of Splunk can now see ServiceNow Incident, Changes, and Problem information. The following screenshot is presenting that IPC information directly for the logged-in Splunk user. This brings ServiceNow IPC information right at the fingertips of the application/DevOps teams that use Splunk to monitor their application/service. Without this integration it can take longer before new Incidents are spotted due to looking at two different tools.

Because all ServiceNow CMDB content AND context is now available within Splunk Enterprise it is now possible to compare content in Splunk with content/context out of ServiceNow CMDB/CSDM. The following screenshot shows how both can be compared with the reason to improve the quality of the ServiceNow CMDB/CSDM. But having this CMDB/CSDM information in Splunk makes it actionable and usable for the application/DevOps teams.

Using Splunk IT Service Intelligence you often want to have control over which Business Service chains or Business/IT chains you want to have updated within Splunk IT Service Intelligence. Most often teams using Splunk ITSI are uploading ITSI Services and/or Entities manually or by file sometimes forgetting to do that in ServiceNow CMDB/CSDM. This works for small installations and small Business/IT chains but it definitely is not going to work if Incidents are automatically sent from ITSI to ServiceNow and there are missing CI’s within ServiceNow. So better to have the CI’s added/updated within ServiceNow directly (central place) and automatically update within ITSI. My solution is not going to delete from Splunk ITSI instead it just gives the user a dashboard that shows what Splunk ITSI Services/Entities are no longer matching with the ServiceNow CMDB/CSDM data.
The following screenshot is showing a situation within Splunk ITSI where the relationship between “app01” and “bs01” is missing, it also misses the “app02” and the relationship with “bs02”. In the after situation, Splunk ITSI is automatically updated by the solution and then having the right Services and relations.

Is the “Splunk Add-on for ServiceNow” still of value?

Yes, we still need the “Splunk Add-on for ServiceNow” on Splunk base as that one is providing the alert possibility to create a ServiceNow event/incident. But we do not need the other feature to retrieve ServiceNow tables anymore. The CMDM is retrieving those tables in context.

What can the CMDM solution bring more than CMDB?

Firstly the CMDM can handle any CMDB/CMS. For now it is prepared to work for ServiceNow but if I get requests for others I will add them.

Secondly the CMDM can handle not only CMDB/CMS objects and relationships. It can be used for all situations where objects (nodes) and relationships are needed like financial (card) transaction fraude analysis or floor/building/city mappings. Please fill in below form and I will be in contact soon.

Availability

Leave your name and email address below and I will be in contact with you.