Likely Wireshark looked into the application data to see the port 80 traffic didn’t include standard HTTP commands, but didn’t do so for the UDP 53 traffic.

Whoa, nicely done! Way to take an idea and a technology and turn it into a service. Well done, Ari. And thank you for taking the time to write about it here!

Hi

About the config. I managed to get some test results from real environment.
And the above configuration is indeed all you need for this solution.

In my example

ip sla reaction-configuration 8888 react timeout threshold-type consecutive 3 action-type trapOnly

this will trigger and create trap when there has been three consecutive timeouts on IP SLA 8888

This command will then create syslogs from the IP SLA traps.

ip sla logging traps

With the command

sh logging | include RTT

you will see these events from local log buffer.

Below is an event when the ping failed to the DC server (via third party SDWAN) from one office router.

023249: Oct 14 2021 04:33:03.272 UTC: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 1
023250: Oct 14 2021 04:33:03.274 UTC: %RTT-3-IPSLATHRESHOLD: IP SLAs(1): Threshold Occurred for timeout

It will not spam more events and instead waits until the ping goes below threshold value.

When that happens, it will generate another log with the recovery.

023254: Oct 14 2021 04:43:41.426 UTC: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 1
023255: Oct 14 2021 04:43:41.427 UTC: %RTT-3-IPSLATHRESHOLD: IP SLAs(1): Threshold Cleared for timeout

You are correct about the monitoring aspect. SNMP could be used along with these to send the relevant events to some outside server.

But there is some hidden potential in this feature. 🙂

Most monitoring is done from data centers to devices/services. (and mostly checking device health itself)
And not from user network to services.

The whole idea of using IP SLA locally would be to give myself data of what happened from the perspective of the office itself. Like I had a computer doing ping/telnet continuously in the office and I could get info what worked and what did not, during the incident. Even when connections are down from the office and the router would be unable to sent any SNMP’s to outside.

The IP SLA reaction will make sure the buffer only contains relevant down/up events without filling the buffer until I get to read them. And there is no need to ask users for ping, nslookup etc. (usually too late as the incident is already over).

I made a set of IP SLA’s that are pinging/testing several targets around the world (via that third party SDWAN).

• data center (our DHCP and monitoring in another region)
• DNS (azure on local region)
• fileshare port (azure)
• google 8.8.8.8

We had a major outage in one site due to that SDWAN breaking partially.

With the IP SLA I was able to tell exactly when different services and connections were down for the office.
In this case

• Internet was available all the time
• Azure connection was down 1-2 hours, no DNS or fileshares. And went up and down couple of times.
• data center connection was down for 6 hours, so no dhcp to the users. The connection actually worked when azure went first down. Then the IPS dropped this connection to recover Azure and left this down until they fixed it later on.

I wouldn’t get this information from the user and it would have been too late when I received the ticket to investigate this.

All our monitoring shows green light on these services as they indeed were up and running, from data center viewpoint. Just the ping to the router was red. But from the viewpoint of the office, they were unable to do anything and different services were partially available at different times.

With time the IPS was able to recover different parts of the SDWAN connections. And now I have detailed list of what was down at what time excatly.

One reason I write here is because after googling for couple of weeks, I couldn’t find examples for what I was looking for. Hopefully someone in the future benefits from this post.

https://xkcd.com/979/ 😉

Hi Ari, glad you’ve enjoyed my content =).

I like what you’re doing. It’s a great use case to spring board to from this article. I encourage you to find lab gear or virtualize some (using GNS3 or EVE-NG).

That said, in enterprise scenarios, you would use SNMP or SDN other automation based tools to validate device health and connectivity. These tools can do “so much more” than mere IP SLA on local Cisco Routers. To that end, I don’t see IP SLA used super often outside for smaller environments or limited test cases.

Still a cool feature though =)

This sounds like perfect questions to test in a lab =). I’m afraid I do not know the answer off hand.

For example: Ping fails 3 times -> create syslog event -> test recovers -> create syslog event. Later on when you come check the routers local logs, you can see when it happened and how long it lasted. Make similar IP SLA for DHCP, ping, tcp ports, DNS etc. and that would collect you crucial information when the problem occurs. Without the need of requesting users for the usual ping, nslookup, ipconfig stuff. Useful when the network problem comes and goes and you are without any data of what actually happened. Other than the useful description from the users “internet is not working”. 🙂

Here is one example of my attempt of achieving it, but I lack test devices so haven’t been able to confirm if it generates the logs in a way I want.

#icmp google
ip sla 8888
icmp-echo 8.8.8.8
frequency 10
timeout 2000
threshold 200


ip sla schedule 8888 life forever start-time now
ip sla reaction-configuration 8888 react timeout threshold-type consecutive 3 action-type trapOnly
ip sla logging traps

Basicly I want to write “sh logging” and see events when the reaction was first triggered and cleared. I’ll post more if I figure this out.

If there is ACL on the router but I configured “ip sla responder”, the router will reply ?

Yes. You could do that. But with 100+ routers, you’d be better off configuring SNMP for network monitoring.