G2… Me too! Introducing LinkRunner G2.

I love my Netscout AirCheck G2, and I’ve mentioned before that one of my favorite features is the wired Ethernet test. It’s so much easier to plug the AirCheck into a port on a wall and have it tell me which switch and interface that port is patched to than tracking a MAC address in a terminal session.

I like the Ethernet test on my AirCheck G2 enough that I’ve been thinking about picking up a Netscout LinkRunner AT to have a dedicated Ethernet/Fiber tester. Thank goodness I didn’t, because I would have been cursing at my terrible timing instead of cursing at the badassedness (this will be a real word one day, just you wait) of the new Netscout LinkRunner G2.

Twinsies!

I was very fortunate to have an opportunity to take the new LinkRunner G2 for a test drive, just weeks before its public debut, and I can tell you that Netscout has packed a lot of improvements into this unit. At a glance, it looks identical to the AirCheck G2 – the two units sport the same chassis. I’m going to make a lot of comparisons to the AirCheck G2, and really none to the previous version of the LinkRunner – I’ve never owned that unit; and really, I think it’s more useful to understand how the two tools differ and overlap.

The LRG2 Home Screen

In contrast to the AirCheck’s customized Linux firmware, the LinkRunner G2 runs Android, and that enables a whole snowball of cool enhancements.

As a network engineer, if I had a smartphone with an SFP slot on it and the Netscout LinkRunner software, I’d have the best tool ever in my pocket, and that’s pretty much what Netscout has done. There is a camera for taking pictures of switches, patch panels, and the copper wiring that you just found that has clearly been some raccoon’s lunch. It also has an LED flashlight for confirming that, yep, the raccoon is living in that hole in the corner (don’t tell me you’ve never run into a data center raccoon… they’re very common in the great white north!), or just tracing cables over ceiling tiles. You can also install your email client on it and, with the optional Wi-Fi/BT/BLE adapter, send those pictures immediately to your manager so they can find out who is responsible for sending pest control. Uh oh, you’re going to have to deal with the critter yourself? Well open up the chrome browser and you can google “nearest pest control”, and then you can watch YouTube while you wait. Nifty.

As a network tool specifically, LinkRunner G2 is all about diagnosing problems on the wire – you won’t be splicing or fusing fiber with this guy, but it will tell you beyond a doubt if you can blame your problems on that wiring. There are three main test pages: SWITCH, AUTOTEST, and CABLE.

The SWITCH page will show you the CDP or LLDP information of the switch the unit is plugged into, the speed and duplex, and PoE info including the voltage and wattage, and which pins are carrying current. From here you can also flash the LEDs on the switch port, with a configurable interval.

There’s also an SFP slot on the unit, so we can do some fiberoptic testing as well. I was initially a little freaked out by the LED behaviour though – have you ever seen an empty copper port with the LEDs on?!? That’s the start of a networking scary movie right there.

“Can I borrow your status LEDs?”

If the unit is plugged into a fiberoptic patch cable with an SFP module in the slot, it can also display the module temperature, and Tx/Rx levels.

Connected Switch and Fiberoptic SFP info

The AutoTest

The AUTOTEST page operates very similarly to the Ethernet Test and AutoTest on the Aircheck G2, with some enhancements. Like the AirCheck G2, The LRG2 displays whether PoE is detected, the link speed and duplex, CDP or LLDP info, DHCP lease details, DNS server assignment, default gateway reachability, and uploads the results to Link-Live – Netscout’s online repository of all of your test information. With Link-Live (included free with either device), all of your testing history is accessible in one place.

PoE Info

Link Info

CDP or LLDP Info

DHCP Info

DNS Info

Default Gateway Reachability

Configurable Target Reachability

Continuous Ping to Target

But the LRG2 takes many of these tests to another level:

• The LRG2 can be configured to request whatever class of PoE you choose – right up to UPOE (25.6 – 51.0 Watts, roughly).
• The TruePower feature can actually load the line; drawing power to tell you the actual wattage and voltage used, which can help point out issues with your switch or the cable.
• This is a big one! The LRG2 allows you to set a VLAN tag and priority. Now you can test actual data transfer on trunk ports and even get a sense of QoS treatment if those ports are congested.
• The delta of the time from the DHCP discover to offer is reported, as well as the delta from the request to the acknowledgment; and the lease length.
• There is now the ability to have the LRG2 run a continuous TCP ping to a configurable target (google.com port 80, by default).
• Pictures and comments can be added to the test results directly from the LRG2 AUTOTEST page – and since there’s a built-in camera, this is really easy to do.
• Wired 802.1X authentication, IPv6, and proxy configuration
• Set a user-defined MAC address for the LRG2 unit. Much easier to find 00:11:FE:ED:FA:CE in those bridge-tables!

Add Pics and Notes right from the AutoTest screen

Red means not good 😦

Ooooh! This makes me happy!

So Netscout has really squeezed in some good stuff here. Next, let’s take a look at the physical cable testing:

This is what you want to see…

When plugged into an Ethernet cable where the far side is unterminated or unplugged, the LRG2 uses TDR to draw out each conductor, points out the estimated cable length (+/- about 1ft) and indicates if there are any breaks (open conductors), split pairs, etc. Here’s one of the Cat5 cables that the previous owner of my house ran out to the garage, but I’ve never been able to get gigabit speeds over:

This is what happens when someone thinks it’s clever to use a single Ethernet cable to terminate both a voice and a data jack.

Well that explains that. There are some accuracy improvements over TDR to be had by using the included wiremap #1 termination (wiremaps #2-5 are available from Netscout).  TDR actually tests per pair, so it can only tell you if there is a problem within that conductor pair. The wiremap allows testing each individual pin.

Without the wiremap, test shows an open pair

With the wiremap, the test is able to determine the specific open pin

By pressing the purple + sign on this page, there are options to send 3 types of tone over the cable so you can use your favorite toning equipment to run around the building looking like you’re dowsing for water while finding out just where that cable really goes. A useful feature regardless of how you look taking advantage of it.

But wait there’s more…

There’s still plenty to talk about here. With the Android OS, the LRG2 gets fast sleep/wake functionality. Just tap the power button to sleep or wake the unit. This helps balance battery drain without having to wait for the unit to restart every time you bring it out of your tool bag. Which is important – because Android can be battery intensive; the LRG2 data sheet says to expect a 4 hour battery life, which is… not much.

Charging via PoE

The good news is that the unit also charges over PoE! But only when the unit is fully powered. So plan to be conscious of your battery life in the field.

There are two big usability features that the LRG2 has that I wish would come over to the Aircheck G2; the first of which are over-the-air updates. Just like your Android smartphone, you can go to the settings menu and check for updates. The unit can then download the update over the Ethernet (though I guess that wouldn’t really be over the “air”) or Wi-Fi connection and install it on the unit. No laptop required.

The second feature is removable micro SD storage. The AirCheck G2 can dump screenshots and session files to a USB drive, but the LRG2 can actually browse the SD card, and files can be copied or organized on the drive right from the File Manager application on the unit. It’s clear to see that Netscout has really focused on enabling more efficient workflows for technicians using this device, looking back at things like the ability to add screenshots and comments right from the testing page. With the Android OS, you can even install your favorite email client on the unit and get your trouble tickets sent directly to the device.

There are a couple of points to note here – the USB Wi-Fi adapter is useful to enable network access on the go. It’s not included in the base bundle, but you could also use a wired connection in that case. In addition, the unit does not have full access to the Android app store. One of the bullet points Netscout is advertising here is the elimination of the security risk of using personal devices… and allowing open access to the app store would make that a debatable point. Instead, Netscout is whitelisting certain apps like Speedtest, iPerf apps, and vendor specific configuration utilities, which are then made available on a Link-Live version of the store. Netscout says for now that users will be able to send a request to Netscout to have an app added to the store if there is a particular one they feel is missing, and there are plenty of useful apps available already.

Speedtest App – over Wi-Fi

So now I’m trying to decide if I can justify the expense of a new LRG2 for my work (yes, they’re making me send the review unit back 😩). At roughly $3000 USD, the unit isn’t a small expense. For me personally, I think the AirCheck G2 covers my use cases more completely, having both advanced wireless testing capabilities and enough wired testing ability for most wireless and even route/switch engineers to determine if they need to call the electrician or cabler (by contrast, the LRG2 Wi-Fi adapter only enables the unit to find a network connection – there is no ability to test over the wireless connection… as of today). It was also tough to justify spending approximately $2500 USD on the AirCheck G2 at the time. That being said, the Netscout team just delivered some BIG enhancements to the AirCheck G2 and I feel like that has made it a great purchase. Knowing how hard the Netscout teams have been working on these devices (look at all of the enhancements today at initial release) and with the LRG2 running on Android,  I would make a big bet that we’re going to see a lot of feature additions to this unit soon enough.

Now if the teams at Netscout could just slap together an AirLinkCheckRunner G2 in the same size and form factor, I’d be smitten. I’ll be fair though – there is most likely a broader audience than network engineers like me for the LRG2 in technicians and installers that need simple test results for a large number of cable drops in a short period of time – and this unit is really going to work well for those teams. I’d love to see the cablers that I work with carrying one of these around.

Advertisement

G2… V2!

Today the cat is out of the bag… check out the full review of the cool new AirCheck G2 features from our secret MFD2 session right here.

I think this is a killer update to the G2 – here’s a quick recap:

iPerf performance testing. (!!!!!) Not only can you now check upstream and downstream throughput against an iPerf server, there’s even a slick new accessory so you don’t have to build your own iPerf box. I’ve been using this on projects and it’s a great way to check only the links you really want to – rather than pulling out speedtest.net on my iPhone where I can’t tell if the internet link is the bottleneck. The accessory looks just like a darker coloured LinkSprinter and can run on PoE (or a couple of AAs), and has a basic web server. Hot damn.

Captive Portal Support. Another big feature, this has made my G2 fully functional where it used to have a fatal weakness – captive portals would render it useless for active association testing. Now the built-in browser makes it possible to fully enter authentication credentials or accept a AUP. Brilliant.

ACL / Authorization classes. This is really AP categorization. Set a big ‘ol flag on that unauthorized AP so you know it’s, well, unauthorized. The G2 will also flag rogues detected on the auto test. Could be handy.

Interferer Identification. I relate this to Cisco’s clean air. The G2 can now detect things like Bluetooth devices, Microwaves, and other sources of interference. The G2 shows duty cycle and which channels are affected. This is not full spectrum analysis, but meet your new handheld interference locator. Shut the front door.

 

 

Interferer Types 1

Interferer Types 2

Packet Capture. Save your session file and now optionally add a PCAP of all the testing you did; then take it away for offline analysis. Includes radio tap headers. Yes please.

Channel Overlap View. Another way of visualizing where all of those APs are camped out, in a view we’re all familiar with these days. From the channels menu, find the overlap button in the bottom left corner to get to the new view. from there, you can tap on any of the parabolic arcs to get an expanded view of the AP BSSIDs that are on that channel.

 

Phew. Let’s take a break right there. That’s a LOT of big features that just got jammed into our handy little tool. Way to go Netscout on adding value to the platform!

If you have MasterCare support for your AirCheck G2, download firmware version 2.0.0 from this link.

You can also find a technical overview from Netscout here, and a recent webinar where Netscout talks with George Stefanick with Houston Methodist hospital about how they’re using the new Aircheck G2 features here.

 

Hours saved by the AirCheck G2

I really enjoyed visiting Netscout HQ during Mobility Field Day 2. They just might have the coolest social media manager in the tech industry (Hi Kendall!!!) and I was excited to meet some of the people on their wireless team behind my favorite tool, the AirCheck G2.  Chris Hinz pointed out a couple of things I didn’t already know about my Aircheck G2. Check out Chris’s presentations at the Tech Field Day page: Netscout at MFD2

In a similar vein, I thought I’d share my two favorite G2 features. I just spent an entire month on a industrial mine site physically recabling fiber optics and replacing switches.

To get an idea of the environment I’m talking about, take a look at these pics:

This is nothing linear about this facility, and I got turned around every day. Thankfully, my G2 has saved me from  hours of frustration on more than one occasion.

I’m almost ashamed to say that I probably use the G2 to trace wires more than I use it for troubleshooting wireless. Looking back at the pics above, you might imagine how difficult it was to follow cables visually. Without a tool like the G2, it would have been next to impossible to determine what was on the other end of a cable. The Aircheck’s Ethernet test has the ability to read CDP/LLDP and give us the name of the switch and the specific port number we’re connected to.

Not to mention confirming that PoE is available, that we’re connected to a Gigabit port, and that DHCP is working. We can see that the G2 has even reached out to Google to confirm that there is a working internet connection, and then uploaded all of the test results to my Link-Live dashboard (a free service).

Besides tracing wired endpoints while I was replacing switches, I was also challenged with some actual wireless reconfiguration. I needed to reboot several wireless access points, but few of the switches in this facility were PoE enabled, and all of the (old 1242s) APs were powered by AC adapters. No easy power cycles from the switch port for me. Now I have the opposite problem from above – I know where the switch is, but there’s no way I can trace cables visually to the AP. AirCheck G2 to the rescue again:

The AP locator gives me an easy “hot and cold” way to find an AP by signal strength. I could do this with a laptop and Inssider or Wi-Fi Explorer Pro, but in this industrial facility, safety concerns make this an impractical approach (and safety rules actually prohibit having both hands full). I need to be able to have two free hands, not to mention that I am wearing gloves, along with a hard hat, safety glasses, and sometimes earplugs.

The G2 simplifies the whole process, giving me a visual indicator as well as an audible beep like a metal detector. I haven’t tried it, but the spec sheet says you can even plug in a USB headset instead of listening to the beep from the unit, which, looking back, would have been useful for working with ear protection. I can also slide the unit into a pocket in order to have both hands free for going up and down stairs and it’s small enough to operate with one hand once I’ve chosen the AP to track down. These are all big improvements over a laptop when I’m not working in a carpeted office environment.

These APs are also al in NEMA enclosures in a facility that has NEMA enclosures everywhere; and most aren’t protecting APs. Suffice to say, it was much easier to locate the APs I needed with the G2. I am even questioning my earlier decision to not purchase the directional antenna for the G2 (which Chris discusses in the TFD presentation linked above). I will gladly fork out the cash for that tool if I ever have to spend a month doing this kind of work again.

Netscout has done a great job with the AirCheck G2. Best of all, the G2 is still a new tool – so expect plenty of innovations and improvements to the G2 feature set to come out of Netscout soon. I can smell good things cooking in the Netscout kitchen…

Wearing the Cape

Looking at Cape Networks after their presentation at Mobility Field Day 2.

There’s a trend in the networking industry of turning client devices into trustworthy monitoring and troubleshooting devices. In the past it was common for users to call the network team to say “the Wi-Fi isn’t working” and the network team would promptly log into the controller/dashboard/APs or other network gear and say “it looks good from this end.” Especially with Wi-Fi, it can be very difficult to understand the client perspective when you’re looking at the network – it’s like the other side of the fence.

Cape Networks is doing a solid job of putting the network engineer on the client side of the fence. Check out the #MFD2 presentation here:

Cape Networks Presents at Mobility Field Day 2

Cape gives us a client device whose entire purpose is to report standard, trustworthy, and consistent performance metrics to the network engineer, in their own language. It’s an interesting idea if you think about it – they’re making a user talk like a network engineer… usually the network engineers are tasked with putting themselves in the users’ shoes.

No more “It was WAY faster yesterday!”

How do I even measure that…?

No more “The Wi-Fi is down everywhere!”

Does that really mean Wi-Fi, or did DNS crash again?

How about a user that can tell you that how much download throughput they’ve had from YouTube for every ten minutes over the last 24 hours, and even as far back as the last 30 days?

How about a user that can give you bar graphs tying RSSI to Channel Utilization, Retry Rate, and BSSID, again for every ten minutes in the last 24 hours to 30 days?

Oh and this user has PCAPs every time they have a “bad” experience, triggered by crossing a performance threshold.

But wait… there’s more! Your superhero user has iPerf stats too.

Icing on the cake: when this user can’t connect to Wi-Fi, the tests still run, and you are still seeing the results in your monitoring dashboard because the user is sending you the stats via their cellular hotspot. There’s also a battery – so the sensor can still send you a notification via cellular when it loses power. In fact, my office sensor has been the first way I find out someone has blown the 2nd floor breaker every time without fail. Damn 100+ year-old buildings and interns with their space heaters. But I digress…

I have junior engineers that can do this for me. But it’s not productive to have them sitting in a remote office running these tests all the time. Plus my boss would make me feed them and give them bathroom breaks. My Cape sensors dutifully sit on the wall, PoE powered, reporting stats 24 hours a day, 7 days a week, 365 days a year, without a dime in overtime pay.

If you’re like me, you’re also digging this dashboard UI. Cape has some killer UI designers. The whole interface is easy to navigate and configure. Having an intuitive and simple UI is an advantage that is hard to overstate – if the functionality is there, an easy UI makes the Cape sensor the Most Valuable Employee you’ve got – but you never have to promote them and search for months for a replacement that will never be as effective and still a pleasure to deal with.

There is some room for improvement. Email notifications get annoying and are often just noise, but this is an issue with any NMS too. I’ve found myself ignoring most notifications, because I’ve learned that those I do check into are often back to normal by the time I log into the dashboard, usually just a couple of minutes. Tuning alert thresholds can help with this. Better yet, I’ve suggested to Cape that something like a mobile app that can give me the green light/red light behaviour without emailing me to death would be a welcome addition – and I’ve been told this is being explored.

Which is another strength – like all of the good cloud based services these days, Cape is pushing out new firmware and features on a regular basis. Both iPerf testing and PCAPs are features that have appeared since I first got a sensor to test in February, and the Cape team is hard at work making improvements all the time.

I’ve had Cape in the office and lab since then, and it definitely beats every infrastructure based NMS I’ve worked with. Keep an eye on these guys and reach out to their team if you’d like to try them out. They’ve been very great to deal with as a company.

In the interest of full disclosure, I won a Cape sensor with free service as a door prize at the WLPC conference in February, and was also provided a sensor as a #MFD2 delegate. I have not otherwise been paid to review their product but have chosen to do so because I have been impressed with them as a solution and company.

 

 

Here comes Mobility Field Day 2!

I’m currently sitting in the Saskatoon airport waiting to board my first flight on the way to Mobility Field Day 2 in San Jose, and I am really excited to be a delegate for the first time!

Side note: as I prepare to spend two days seeing what the best in the mobility industry are up to, I’m disappointed to be sitting at YXE airport using my hotspot because the Wi-Fi is currently completely inoperable. It had been working very well on recent trips. Oh well.

First of all, I’d like to send a little shout out to the Tech Field Day crew – they have been awesome in helping us get ready to travel to San Jose and participate in this fantastic event. They’ve clearly done this before and know all of the steps to make sure everything goes as smoothly as possible. Thank you TFD team!

Tune into MFD2 on Tuesday, July 25 here. Dont forget to follow the hashtag #MFD2 on Twitter during the event to catch our up to date thoughts, and mention me (@CdnBeacon) or any of the other delegates if you have any questions you’d like us to ask!

Here’s what I’m looking forward to:

Mist Systems

I’ve had the chance to play with the Mist APs a little bit recently, and I’ve been impressed. It was very easy to get an SSID spun up, and what was really neat was how easy it was to setup their virtual BLE (Bluetooth low-energy) beacons. It took me longer to find the floor plans for my house (and I already knew where they were) than it did to set up the beacons and use their app to watch myself wander around the house. I wasn’t terribly familiar with Mist’s solutions before this, so I’m really looking forward to going in depth. Keep your ears open for things like “blue-dot experience” and lots about client experience analytics.

Mojo Networks

Mojo has gone through some reinventions over the past years. I was more familiar with them as Airtight, but they have a new focus now. I saw them at WLPC and their mojopackets online PCAP analyzer it pretty neat. I’m very curious to hear about their plans for Wi-Fi on open hardware. We’ve heard that idea before, let’s see if someone can make it a feasible reality for the enterprise, and in my case, industrial use cases as well.

Cape Networks

I was fortunate to come home from WLPC with my own Cape sensor, and I’ve had it running in my office ever since. Cape has a great UI, making it really easy to see the important metrics – how is the Wi-Fi working? I think it’s narrow-minded though to think of the Cape sensor as a Wi-Fi sensor only, because it’s really good at letting me know when the network isn’t the problem. Cape has some great potential and now that I’ve spent some time with it, I’m ready with some ideas for tweaks and improvements – bring your notepad, Drew. 

Nyansa

Nyansa has been doing some really interesting work in gathering huge amounts of Wi-Fi client behaviour data, and turning it into informative insights. This is another one that I am looking forward to learning about a bit more in depth. With so much client variation in our environments, I really love the idea of being able to identify, for example, that any iPhone 6 running iOS 9 has a bad habit of sticking to an AP well past when it should have roamed. 

Netscout

One of my favorite tools is my Netscout Aircheck G2. There is a lot of talk in the Wi-Fi industry these days about how difficult it is to get relevant data about Wi-Fi and client performance from the wireless clients themselves. My G2 is great for giving me data from a client perspective, and sometimes there is no substitute for a handheld tool. I’m still a network engineer as much as a mobility engineer, and I admit I’m trying to find a good reason to buy a LinkRunner AT. Netscout’s tools are great for giving me the answers I need quickly. 

On-premise IT roundtable

Rumor has it there might be a recording of Gestalt IT’s On-premise podcast. I’ve been enjoying these episodes since they started as a “just the right length” dive into various issues and technologies for our industry. Keep an eye out for an episode from MFD2 on iTunes, or the website here.

See you in San Jose!

ESS 9 is here: Demo with Jussi!

We had an excellent opportunity last week at Cisco Live! to visit with the team from Ekahau and hear about several things that they’re working on, including some significant improvements to ESS in V9 (which has now been released – if you have an active service contract, go download and install it now!).

The team from Ekahau has a reputation for really being a part of  the Wi-Fi engineer community, and really listening to what the community is asking for, and ESS 9 is a prime example. Ekahau has spent over a year working closely with Wi-Fi experts to add density and airtime utilization calculations that are grounded in solid RF theory.

Check it out below, and for more info, see the Ekahau blog here.

Changing Wireshark Link-layer Header settings on Mac OS

This is one of those quick posts aiming to save me and (maybe you) some time the next time I forget this.

On my Mac, I use Wireshark primarily to capture Wi-Fi traffic, in monitor mode. I want to see the Radiotap and 802.11 headers. Usually I leave Wireshark set this way.

On occasion, I actually use Wireshark to inspect higher level traffic – I want to see the IP addresses and TCP/UDP ports etc. I might be troubleshooting an issue and am using my Mac as the client trying to recreate the issue – so I don’t need monitor mode for that. Simple enough – turn it off in the interface settings (Find this button on the Main toolbar   to access the menu, then scroll to the right to find the Monitor mode drop down and make sure your Wi-Fi interface has this disabled):

Then just set the Link-layer header back to Ethernet, just like your other interfaces:

Except “Ethernet” isn’t an option. I could’ve sworn that’s what it is set to by default after install…

I can’t believe this still trips me up every few months. I spent half an hour the other day scratching my head, when the trick is simply to restart Wireshark. Close it entirely, reopen it and voila:

Ethernet is back! Also, the 802.11 options have disappeared because we’re no longer in monitor mode. Now I can see Ethernet, IP, and TCP/UDP headers again:

In comparison to capturing 802.11 frames in monitor mode:

I keep forgetting the need to restart Wireshark for the Link-layer options to change #facepalm.

Note: you also need to restart Wireshark after enabling monitor mode before the 802.11 options will show up in the Link-layer header drop down option.

Or maybe it’s just me. I’m confident that I’ll still forget all about this post next time I try to show a University Computer Engineering class how many packets it takes to load the Facebook home page.

It was 781 (including DNS lookups and a couple of retransmitted frames), in case you’re wondering…

Get your 802.11eh patches!

Back from WLPC 2017, where the 802.11eh stickers and patches were more popular than I expected.

If you’re coming to Cisco Live! 2017 and would like one (or some or dozens), please enter your info below and I’ll place an order to bring with me.

Cost per patch is $6 USD, which is just enough to cover my cost.

Stickers are available via Sticker Mule, here

PING! Not always what you think! – Meraki Wireless troubleshooting

I’m quite fond of the Meraki dashboard. I’ve seen firsthand how it can enable lean and low-skilled IT departments to manage more of their own networks themselves. The dashboard GUI makes it easy to find status and troubleshoot at a basic level, but it’s still important to actually understand what is going on under the hood.

Here’s an example. If you’ve ever seen the Meraki dashboard, you’ve probably seen the Ping tool on every client status page. Here, a Meraki AP is successfully Ping-ing my MacBook Pro:

Pretty straightforward. Ping the client, client responds, client is online and working, right?

If you have a Meraki Security Appliance, you may stumble across this little note on the Addressing & VLANs page:

Wait… Ping is based on ARP? What happened to ICMP?

We may have jumped to conclusions here. As it turns out, Meraki is not using ICMP like most of us would assume. Here’s an example of a few PCAP frames of that same Meraki AP ARP-Ping-ing my MacBook Pro:

Notice this is a directed-ARP; the Meraki AP (MAC 13:da:90) is sending an ARP request to the MBP (MAC 91:75:d8) rather than sending a broadcast. That is, the Meraki AP already knows the MBPs MAC address. But the ARP response tells the Meraki AP that the MBP is alive, and online – just like an ICMP Ping.

This brings about an interesting question. We network engineers often use Ping as a way to confirm that the network is working. A successful Ping means that routing, IP addressing and the physical path are all functioning correctly at layer 3. But if we’re doing a Ping at layer 2 with ARP – would we be wrongly assuming all is well when we get a response, just like with ICMP?

There is definitely some potential to make incorrect assumptions here. In fact, even though that screenshot above of the Meraki AP Ping-ing my MBP has a loss of 0%, at the time, my MBP had an incorrect IP address and was not Ping-able by other devices in the network (well, via ICMP at least). Here’s the PCAP’d ARP frames from the same time as that Ping output:

Almost identical, except that both the ARP request and response are from 10.11.3.1, when the subnet is actually 10.11.30.0/24. However, the client is still responding, albeit at layer 2, and that’s good enough for the Meraki AP.

Now, I do think this is one of those things where the vendor has made an odd decision to label this as a Ping without being clear about what is actually being done, but it is after all our responsibility as the network engineer to know what we’re looking at. There are similar examples where traceroute can use UDP or ICMP, depending on the OS, and now you know that sometimes Ping is ARP instead of ICMP.

Here’s the relevant documentation:

Meraki Ping Tool

ARGGHH ARRRRRRRP!!!

I run into this issue several times a year with a certain local DSL/Fiber service provider and ASA firewalls. Sometimes it consistently occurs after an outage, others it is seemingly random. This is the relevant debug ARP input (modified for confidentiality):

?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 216.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending

What makes this a REALLY bad thing is that 16.197.160.254 is the ASA’s default gateway. No internet access for you.

Unfortunately, I have little (no) visibility into the provider settings, but I can hazard a guess that there is some sort of spoofing protection in place. Sometimes a reboot of the ISP modem will fix the problem, but often times we have to call the ISP and, while trying to explain ARP to tier 1 support should NOT be too difficult, it is typically an exercise in frustration. Not surprising, often we are asked to plug a laptop directly into the modem with the ASA’s static IP programmed on its NIC, which works, and causes the ISP to cheer “NOT OUR PROBLEM!” Of course, if I plug a laptop directly into the Provider1 interface on the ASA, it gets an ARP reply right away and communication to the “gateway” also works just fine. Ultimately, I have not been able to find an easy, repeatable fix from the side I have control over (the ASA), but sometimes the ISP clears an ARP table to solve the problem.

Something similar occurs from time to time with this ISP with NAT/PAT IP addresses that are NOT the ASA’s interface address. In this case, we can PCAP traffic leaving the ASA with the appropriate IP and MAC towards the ISP, but the ISP will never forward the return traffic to the ASA – the gateway doesn’t create an ARP entry for that IP.

In this case, an easy fix is to temporarily change the ASA interface IP to match the NAT IP. This causes the ASA to generate a gratuitous ARP and suddenly the return traffic gets delivered.

This is certainly different from the first case, where no amount of restarting or GARP seems to convince the ISP gateway to reply to the ASA ARP request for the gateway’s MAC, but I suspect the cause may be related.

I’m currently waiting for the ISP to determine if an engineer who actually knows how to log into the gateway router and look at an ARP table does indeed exist, or whether I’m more likely to watch a unicorn run a red light on my commute home. In the  meantime, if anyone can explain what this ISP is doing to cause this behaviour, I’d love to hear it.