#arp cache

If you’ve ever wondered how devices communicate on a local network, ARP is the answer. ARP, or Address Resolution Protocol, is a key component of networking that allows devices to communicate with each other using unique MAC addresses. In this post, we’ll dive into the details of ARP, how it works, and how it helps to facilitate communication between devices on a local network. We’ll also cover…

Let's say you have a great job and things were working smoothly until you get a ticket from someone saying they have no Internet suddenly you get another ticket for the same thing in a different part of the building. Suddenly you have about half the people in your organization stating they have no Internet. What do you do?

You check all your settings and make sure the ISP is still up, check your hardware, check your physical connections and if all checks out you could possibly be a victim of ARP poisoning or you could be experiencing some side effects of malware that enables a proxy when there is no proxy server to be used in your LAN/WAN. 

At my office we had an issue as mentioned above suddenly people would have problems with Internet access after we purchased new hardware and moved to a new ISP. When the issue first became present I couldn't find a rhyme or reason for the issue. My boss and his boss and other employees were getting PISSED off and rightfully so, they couldn't do their job without a continuous Internet connection. I followed the standard procedure of checking all hardware, connections, ISP etc... well I called in to our firewall manufacturer to make sure there was no defect on their side and the agent said it sounds like you have ARP poisoning, now this requires explaining. I informed the agent that the Internet connections came back after clearing the ARP with netsh int ip delete arpcache and using netsh int ip reset and our firewall LOGS did not show anything abnormal when the drops occurred.

I had so many theories why the Internet drops occurred but I wasn't really buying in to the ARP poisoning solution I should have been seeing the signs while using WIRESHARK. I could not find the culprit. So time passed found a switch that was sending excessive broadcasts I reset the switch and problem solved, or so I thought. A month later the issue came back, reboot hardware didn't do the trick anymore. I decided to divide networks and split traffic to separate ports instead of having everything go out one gateway. That tacit provided relief for about two weeks, the long term benefit was better QoS and load balance, but the issue came back and it was the same group of people that had issues last time. So this time I set a rule in our logon scripts to clear the ARP cache at logon and run a scheduled task every hour clearing the ARP Cache, again this worked for a few days, a week at most. Then I broke out WIRESHARK again and let it run for a few minutes versus a few hours (limiting the amount of logs). After I pulled the packet captures I used the filter eth.type==0x0806 and I looked for anything suspicious and I saw it, the missing link. There before me was the line "Duplicate IP Address Detected" It so happened the IP address detected was that of my firewall Gateway for our employee subnet.

Naturally I tested the theory by locating the hardware via the Mac address, turned out to be our video conference system, and unplugged first. Ran WireShark and no conflicts, I changed the IP address and ran it a second time, no conflict. Therefor we have found our culprit, it wasn't a rouge machine, malicious employee, or malware, it was a conflict of IP. The prior IT Manager did not document the IP address information anything in that building so everything was a shot in the dark. My fault was not documenting or attempting to document the IP addresses of the machinery however, the point is I found the resolution using free tools. I will say that there was some help on the Internet but I couldn't really get much assistance, because it is hard to explain these things when you are stressed and have no energy.

Summary; if you have random disconnects from the Internet in your office, do a virus scan first! Check your physical connections, hardware and software. Then if you are still not getting outside check your IP addresses for duplicates (or conflicts) and use WireShark it is a helpful tool that is free, remember to use eth.type==0x0806 as your filter.

Below are some helpful blogs and tips I used in my research with my problem.

http://www.ostalks.com/2011/11/04/arp-poisoning-and-detection/

Problem

Following the installation of a Cisco ASA our DHCP server on the same broadcast domain started listing BAD ADDRESS entries in the database when clients were trying to renew or get a new IP address. The result was that no client could could get a valid IP address and was reporting IP address conflicts.

Details

DHCP Server: Windows 2008 Server.

DHCP Client(s): Windows 7.

Firewall: ASA 8.4(2).

Cause

The DHCP process is as follows:

The client issues a DHCP DISCOVERY broadcast.

The server responds with a DHCP OFFER broadcast for the client.

The client issues a DHCP REQUEST unicast to the client.

The server responds with a DHCP ACKNOWLEDGE unicast to the client.

Finally the client issues an ARP request to see if the IP address is not in conflict with another client.

The problem is that the ASA responds to the ARP request making the client believe there is a conflict when in fact there isn’t. This makes the client issue a DHCP DECLINE to the server which in turn marks the address as being already in use (BAD ADDRESS).

The problem is further worsened as the client starts over the DHCP process again, the server issues another IP address, the ASA responds to the ARP request and the next address is rendered un-usable in the database. This process continues until all of the IP addresses in the DHCP pool are marked as BAD. No client on the broadcast domain is able to receive a valid IP address.

Solution

The cause was discovered/confirmed by installing Wireshark on a client machine and monitoring the DHCP process. It was found that the ASA was responding to ARP requests for IP addresses for which it did not use.

The ASA was configured to proxy ARP requests on the inside interface.

To disable proxy ARP on the inside interface the following commands were issued:

ASA# sysopt noproxyarp inside