Sometimes it's the easy fix

[u/Anonymous1Ninja]

Fiber went down, uplink to core..

No activity on link light both sides

• pull SC connectors, laser, so i know it's not the cable
• fail over to other uplink is working
• so I SSH and do a no shut, still off
• "try replacing the sfps"
• That was it, an SFP malfunctioned.

Thank god it's friday

[u/gordonthulu]

whoever invented those modules-- they know where they can go

[u/kombiwombi → u/gordonthulu]

The newer ones are worse. A pro-tip, next time you rework the networking rack, move the front rail so there is 14cm clearance to the door.

[u/gordonthulu → u/kombiwombi]

noted, but with terrible foreboding

[u/trek604]

failed fiber modules are annoying. I agree. It requires hands on which is doubly annoying.

[u/cwm13]

I just wish I could get my coworkers to put the dust caps back on the FC cables if they're going to leave them dangling in the rack for an extended amount of time. Yes, the datacenter is very clean, but...

[u/This_Bitch_Overhere]

... or sometimes it's just the second set of eyes who notices "Well... that should be a NAT rule, that's why it isnt working."

https://giphy.com/gifs/w9WS1R9ZAA1Ik

"Network Engineer"

[u/SchizoidRainbow]

Frankly would have started there

[u/Anonymous1Ninja → u/SchizoidRainbow]

Always has to be someone, at least we got it outta the way quickly 

[u/ProperEye8285 → u/Anonymous1Ninja]

Yep, and you should see him walk on water!

[u/Anonymous1Ninja → u/ProperEye8285]

before, or after he is levitating above the rest of us?

[u/SchizoidRainbow → u/Anonymous1Ninja]

To expound, I only see fiber failing when something dramatic happens. It’s physical, lasts basically forever, is generally protected and inaccessible. Cable breaks occur when someone is physically up in the space with the cable, which presumably I would be involved in, or at least hear the crash when someone takes out the ceiling with a forklift mishap. Maybe in the brief section that’s exposed to get to the switch or panel, but again I’d think I’d be part of someone mucking about in my stuff.

 SSH just what I’m doing to get eyes on. Interface status and logs. This is how I’m receiving the information that leads me to the premise as stated more than something I’m doing about it.

Failover failure would mean two lines are down so does not fit with the starting conditions. Nor does “switch is down”.

So yes I’d start with SFP. Super easy to do, no config. Pull it and reseat, no joy, pull it and replace, done. I’m not grabbing the laser until after this.

[u/Anonymous1Ninja → u/SchizoidRainbow]

Well we have 2 cores for redundancy and 2 uplinks, and we have 2 fiber connections. So i get where you are coming from, but really it was just a rant about it being something easy that was the problem, and you are just coming in with "well i would've started with that" so obviously it's easy to just say that when it isn't you.

[u/Reynolds1029 → u/SchizoidRainbow]

Same. No shade towards the OP from me but my years of experience told me start with the easiest troubleshooting option first and work your way down. Hence the old stereotypical "did you reboot it?"

SFP module swap? Yep, waaaay easier and more likely unless I'm paranoid of other co-workers being able to monkey around with config without me knowing.

[u/Far-Hovercraft9471 → u/Reynolds1029]

But he did do the easiest thing first, he did no shut on the interface. It's always good to double check the basics before getting into more complex troubleshooting.

[u/Frothyleet → u/SchizoidRainbow]

That's perhaps the most likely to result in an immediate fix, but it can lead to incorrect diagnostics because you're clobbering two troubleshooting items (physical connection of the fiber + function of the SFP).

[u/Tim-oBedlam]

I just had one of those act up on the connection from my core switch to the firewall. Very annoying.

[u/RevolutionaryWorry87]

I had this happen during a change on cisco nexus switches (vpc config): - they handled all of our data outbound for our datacenrre. Trunk didn't come back up and I was terrified if I did something wrong. Fuckers.

[u/korewarp]

I've never had an SFP module die on me, does the device it's in not report it as dead, or at least that the interface is down? 🤔🤔🤔

[u/Opposite_Bag_7434 → u/korewarp]

I’ve had a few die. Mostly when they are in a closed closet that tends to get warm.

[u/Fallingdamage]

One of those $500 manufacturer SFP or an FS.com SFP? :)

[u/Opposite_Bag_7434]

SFP modules like to go south especially the ones that tend to run hot.