Work Life: Two ports in the sea

An area school reached out because they were having some very bizarre network issues. They had replaced several switches on their network and things seemed to be going fine - for a while. Soon, they noticed segments of the network were crashing or coming to a grinding slowdown.  First inclination is to blame and look for a network loop in effect.

For those that may not know, a network loop happens when you take a network cable and plug both ends into the same switch. For example, you plug one end into port 1 and the other into port 8 on the same switch. Much like an echo between speakers and a microphone, the network traffic coming out of one end feeds into the other end, which then feeds it back to the first end. Lather, rinse, repeat. With microphones and speakers, we hear the loud squeal of feedback. On a network, traffic loops in an ever-increasing roundabout fashion, bringing things to a standstill.

The same thing (or similar, anyway) can happen if two network wires from one switch are plugged into two ports on another switch. For example, Port 3 and 4 on one switch are plugged into ports 1 and 4 on another switch (the actual port number do not matter). This is usually called "spanning tree" since it is between two switches, though the term can also be used within a single switch.*

So, you with me? Okay, here's the situation then:

When the tech plugged in a new switch via fiber, the network would come crashing down. Okay, we thought, must be a loop somewhere in the "new" network since things worked fine before hooking up the new switch. Except for one problem: it happened even when the new switch had NO OTHER connections attached - just the fiber from the main network switch. Unplug the fiber, and the network settled back to normal.

Okay, let's see what else we have then. We head to the main network switch (also referred to as the core switch) to see what we can figure out. As we looked over things, I had to climb a ladder in order to get to the core switch because it was mounted high in the network closet's rack.

After a few moments, I realized the problem: they didn't have a loop so much as they had two network lines plugged into the same port. Wait. What? How could they have two wires in the same port? Allow me to demonstrate. Note: The image below is NOT the actual switch in this post. I recreated the scenario using my own switch.

Now, you may be saying, "Those circled wires are not in the same port!" While that is true, physically, it is not true logically. Notice on the left, the port number is 23. On the right, it is also 23. In most switches that have CAT5 (left) and fiber (right), they share certain ports. In the image above, ports 21-24 are "shared." This means you can EITHER have a CAT5 wire in a port number *OR* you can fiber in the port number. At least, from a logical sense. In a physical sense, you can plainly see there is nothing stopping me from plugging a wire into both ports. The switch in this case sees both as perfectly legitimate and sends/receives traffic down BOTH of the Number 23 ports. Much like a network loop, traffic spins out of control as the switch sends and receives data down both paths.

That is what happened at the district. Now, seeing the example above makes it look like the problem is obvious, right? Well, in the example above, it is obvious. In the real world scenario, as I mentioned before, the switch was mounted up high. Additionally, that switch had the port numbers on the TOP of the ports, not the bottom. So, the tech could not see that the fiber ports were shared with ports 23 and 24 (that switch only had two fiber ports).

Because the tech could not see the numbers, the fiber was plugged into Port 24.  There was also a CAT5 wire plugged into Port 24. So long as the fiber was not plugged in on the other end, everything was fine. As soon as the fiber made the connection in the other switch, everything went crazy, as by now you might imagine.

The moral of the story here is that we can't always tell the exact nature of the problem due to the actual physical parameters of the environment. In this case, the tech could not see the numbers on the ports, thus not realize wires had been plugged into the shared ports on the switch.

Sometimes, it truly does take a different set of eyes to help solve a problem.

*Note: Some switches can be configured to handle such arrangements. This can take the shape of VLANS (virtual local area networks) or spanning tree detection/prevention. You may have heard these referred to as Layer 2 or Layer 3 switches.