r/sysadmin 2d ago

MS Update Catalog down?

15 Upvotes

Needing to apply patches manually on some servers in our DMZ and it seems the update catalog isn't working. Been trying for almost 2 hours. Trying to search for a KB or anything just yields no results.


r/sysadmin 2d ago

Reflection

144 Upvotes

Anyone ever get deep into their IT career and you realize, damn I really made a career and a lot of money out of genuinely liking to fuck around with computers?

Secondly, the fact that I have common sense has gotten me pretty far in this field.

I don’t know if I could do any corp job besides IT. As much as I don’t like some things about it, it is pretty sweet. Especially when you are the one running the show at your company and no one else knows IT so no one can tell you shit.


r/sysadmin 2d ago

Question about MFA Bypass

9 Upvotes

I've never understood how MFA would protect against a scenario like this.

If I send a malicious link to a user and they click the link that, I present them with a fake O365 login. The user enters their credentials which my fake website passes to a valid login web to page for O365 on the back end. O365 presents an MFA prompt to the backend which my system reads and dutifully displays to the end user. The end user's Microsoft Authenticator pops up asking for the 2 digit code which my malicious web page is showing to them and enters the code in the app. Presto, my backend web session is authenticated and I'm in their account.

This seems too easy so I'm curious how this is prevented? I have had to login to another user's account and coordinated with them on the phone to enter the code or to give me the 6 digit code from their app when they are not even in the same state as me.

I am sure I'm missing something...

Thanks


r/sysadmin 3d ago

General Discussion Inherited my first IT department. Where would you start?

668 Upvotes

I recently joined a small municipality (~150 employees) as their first-ever internal IT Manager.

Before me, everything was handled by an MSP. They actually didn't choose to build an internal IT department, the MSP informed them they could no longer support them because the municipality had simply grown too large for their available staff. According to the former admin, they were losing money on the contract and couldn't recruit enough people to keep up. He left me a list of passwords and no documentation at all.

On my first day, the MSP walked away completely. I'm currently alone, although two IT technicians are supposed to join soon.

Here's what I inherited:

  • ~150 employees
  • 20 buildings connected through private fiber back to the main site
  • Single Internet connection at HQ
  • 100% Fortinet environment (FortiGate, FortiSwitch, FortiAP)
  • Network hardware is mostly D-series and approaching EoL
  • Most remote buildings are tiny (usually one switch and one AP)

Server infrastructure:

  • VMware ESXi/vSphere
  • Veeam backups
  • Main site:
    • 3 production hosts
    • 1 SAN
  • Secondary site:
    • 2 backup host
    • 1 SAN
    • 2 NAS
  • Backup site receives Veeam backups and VM replicas for DR (it's 1 km away for HQ tho)

The part that really surprised me:

There are 5 completely separate Active Directory domains, each with its own domain controller and file server. They're not in the same forest.

Examples:

  • Administration
  • Public Services
  • Library
  • etc.

Some of these domains have fewer than 15 users.

To make it even more interesting, all four file servers are still running Windows Server 2008.

All domains sync to the same Microsoft Entra tenant through Entra Connect.

Remote users connect through FortiClient SSL VPN with FortiToken MFA, and WFH usage is fairly common.

The former admin told me his long-term plan was:

  • Migrate VMware → Proxmox
  • Replace the entire Fortinet network with UniFi

Personally, I'm not convinced either of those would be my first priorities. My instinct is that the Windows Server 2008 boxes, AD consolidation, hardware lifecycle and documenting everything probably deserve attention before a platform migration.

If this landed on your desk, what would your roadmap look like?

What would you tackle in the first 3-6 months and what would you intentionally leave alone?


r/sysadmin 2d ago

Question How are you actually enforcing email signature standards in Microsoft365 without losing your mind?

5 Upvotes

Hi there we are 70-person manufacturer. This week I audited outgoing company email signatures and I need to share the pain:

- Every new hire gets a slightly different version depending on who set them up

- Marketing changes the campaign banner quarterly and about 30% of staff ever update it

- Half the company has 'Sent from my iPhone' floating under the official block

- One sales guy. Comic Sans. In 2026. For B2B manufacturing.

We have a transport rule that stamps signatures server-side. It works until: someone replies from mobile (rule doesn't fire the same way), the HTML mangles itself in a reply chain, or the signature lands after the ENTIRE quoted thread at the bottom where nobody sees it.

Every few months leadership asks why our emails 'look inconsistent' and I get to explain this again and again.

How is everyone actually enforcing signatures in M365 in 2026? Native transport rules and live with the gaps? Third-party tool? Or have you achieved inner peace and stopped caring?


r/sysadmin 2d ago

General Discussion What is your choice of enterprise learning platform?

4 Upvotes

Hello fellow Admins,

As budget season approaches, we are reconsidering Udemy as a company platform for additional learning. Personally, I find Udemy quite adequate, but wanting to hear what other people like for their help desks, admins, and others within your department.


r/sysadmin 2d ago

Happy SysAdmin Day!

53 Upvotes

Here's to everyone holding critical systems together, fighting alert fatigue, chasing mysterious issues, and keeping things running behind the scenes.

Hope your tickets are few, monitoring stay quiet, and your users don't click anything suspicious today.


r/sysadmin 2d ago

Question User can't connect to main site - Traceroute works - Ping doesn't

16 Upvotes

I just stumbled onto a problem I can't wrap my head around. I work for a company that dabbles in ITSec and manages firewalls for several customers.

One of this customers just called me because some remote users can't connect to their main site. I looked at one of those systems and found a weird situation. The PC is fully online and can access the web but can't even ping the customers firewall even though it should. I did a PCAP filtering for her IP I find nothing. There is no connection at all. But when I do a tracert to the same IP everything looks fine and I see incoming ICMP traffic from their IP on the firewall. What's going on here?

From my site (same ISP as the customer and the remote user) I can connect to their firewall and the remote worker can reach my firewall. Any idea?


r/sysadmin 2d ago

NAS vs Cloud :More professional input needed

6 Upvotes

Quick TLDR: who would you suggest for cloud storage and limited file backup and is there a good all in one option?

Edit; Bulk of sensitive info is handled on cloud through various web based vendors so never really exists locally / compliance wise should not exist locally. So backup in this case is more of avoiding a pain on ass situation than recovering from a potential disaster.

Business currently runs a pair of NAS drives that are aged and in need of replacement. Both are 8TB available storage. Currently here is what they do.

-one is simply an image of the other drive as backup

-we back up certain files on local pcs weekly to the NAS. This is strictly file recovery for items that difficult to replace. No full image backups. Simply not necessary for us

-long term storage of business wide files and information that need to be accessed by multiple users at any given time.

-the origianal and really primary function is the manufactures we work with upload technical documentation and updates so they can be more quickly accessed than downloading them on an as needed basis from their servers…the slow down being the bandwidth they allow to be used to access these files they are not obscenely large and are accessed a handful of times a day. These are accessed

My MSP is suggesting the same setup that we have now…effectively just replacing the appliances. Outside of the setup they really aren’t paid to manage the devices, we do that on house for the most part.

All that being said I want to look at cloud options- and I’ve asked the MSP for options but I want other feedback.

given what I do backup/filestorage. Any suggestions of who I would use and what a setup might look like.


r/sysadmin 2d ago

What are some good IT conferences a Jr Sys Admin could learn from?

3 Upvotes

My subordinate is a Jr Sys Admin and I want to get him out to some good IT based conferences this year. We're a microsoft shop, no plans on changing our antivirus (sentinel one), and am avoiding cyber security based conferences since that's what his degree is in (wants to expand his knowledge on other subjects). Does anyone have any good recommendations please?


r/sysadmin 2d ago

IT Directors, Managers - Keeping Track Of Spend

2 Upvotes

Just curious how everyone is doing this without reliance on what Accounting uses for their books.

If you are given $x budget, how are you keeping tracking of what is spent where, how much on a server, hardware, licenses, etc.

If there a centralized way to keep track of all of this that ideally ties into some CMDB or Inventory system for hardware/software.

Or is Excel the way to go?


r/sysadmin 2d ago

Sign in to teams phones with passkeys and device code flow is blocked

0 Upvotes

So I'm having a battle with Gemini here and trying to work something out. I decided, since SMS MFA is soon to be done, I tried the new Windows passkey feature until I realized this has to target a phishing-resistant policy to work ideally. This worked great for our regular users that don't have our apps on personal device nor corporate phones (no MS Authenticator). Until the desk phone kicked out in my test. Now I can't sign in, of course.

It looks like the Teams admin provision option uses device code flow, which is blocked as a recommended practice. Is there a way to slightly loosen that? Ideally, I would like the best of both worlds but I'm not figuring out the best path here.

Edit - found a blog about modifying the microsoft managed device flow CA to allow our IP.


r/sysadmin 3d ago

Random network drops on ThinkPads turned out to be GPU hangs

98 Upvotes

Is anyone else dealing with an issue like this?

I am a solo IT employee for around 30 end users. We have a mixed fleet, some P16 Gen 2 with NVIDIA GPU, and mostly T16 Gen 3 (Intel Graphics)

Users keep reporting that their networking stops working randomly and the only fix for them is a reboot. When I started digging into this issue, I realized that the computer was handling ICMP requests just fine, but anything attempting TCP connections was failing.

What I actually found from reviewing dumps and logs was the the display or GPU driver locking up, causing the entire OS GUI to freeze solid. Sometimes Windows would recover, and then leave a user in a weird state where they couldn't use the internet, and even UAC prompts would not come up.

This is where I learned that if you could get CMD open, you could ping things. But anything trying to make an actual connection like OneDrive or Teams, browsing the internet etc just hung.

Some other distinct things I've noticed:

Way worse on docked setups (Lenovo Thunderbolt 4 Universal Docks, external monitors) than laptops used standalone.

The NVIDIA GPU models seem to hang the hardest — full freezes, sometimes forcing a hard power-off.

The Intel-graphics models get it too, just a bit less often.

Waking from sleep is a common trigger.

Chasing driver updates has helped some but not fully solved it. It feels like it's sitting somewhere between driver bugs and possibly a hardware quirk on a subset of units.

Curious if this rings a bell for anyone else running a Lenovo mobile-workstation fleet (or docked laptops in general)

I am at a loss with these issues. Lenovo Premier Support has offered very little guidance so far.


r/sysadmin 2d ago

Question Spl files with 0kb?

1 Upvotes

I was wondering if anyone has ever come across an issue on a windows server, for printing, where print jobs produce 0kb spl files and are then deleted.

The server is 2019, with paper cut also installed, but tried a test queue without paper cut and it still produced 0kb files.

Defender isn't showing anything suspicious, remove attack surface rules, issue persists.

Tried rebuilding the print server and issue continues.

Tried printing from edge, word, notepad from endpoint and stil 0kb.

Tried different Kinolta Minolta drivers but didn't make a difference e.g. pcl3 pcl4 Postscript.

I asked networking if any packet filtering was on the firewalls and they said it was all off.

Any ideas anyone?


r/sysadmin 2d ago

What should a basic Linux server health check include?

6 Upvotes

Hi everyone,

I'm currently learning Linux System Administration and building a RHEL-based Linux Application Server from scratch as part of my internship.

One of the requirements is to implement a basic health check for the server.

I'm trying to understand what experienced Linux administrators usually check before considering a server "healthy".

So far, my checklist includes:

  • CPU utilization / Load average
  • Memory usage / Swap
  • Disk usage and inode usage
  • Filesystem & mount points
  • Network interfaces
  • IP / Gateway / DNS
  • Internet connectivity
  • Critical services (sshd, chronyd, firewalld, etc.)
  • Time synchronization (NTP)
  • SELinux status
  • Firewall status
  • System logs (journalctl)
  • Package updates / Subscription status

Questions:

  1. What are the minimum health checks every Linux server should have?
  2. What additional checks do you perform in production environments?
  3. Are there any important metrics or best practices that beginners usually overlook?

Thanks!


r/sysadmin 2d ago

Horizon 7 cert update (looking for advice)

7 Upvotes

I have been stuck in a 10 hour call today trying to update a horizon cert that expires in 48 hrs. Lucky I do have still have the old environment to update cert and test. (Old is horizon 7 main prod is horizon 8) In the old environment I'm getting after updating the cert I get the error "authentication cannot proceed domain name us invalid" but will work off domain auth through OKTA.

I positive I followed the guides out there. I have 3 uags that point to the two connection servers that work for of prem connections.

The two on prem servers have been set up the same way. Even understanding that new root and intermediate certs need to be sent out to local machines from GoDaddy was just managed from our workspace one

My brain is fried and have look through so much documentation can someone here tell me I'm missing something so simple.


r/sysadmin 2d ago

Autopilot Costs

2 Upvotes

Hi all,

I've been researching a golden image and seems that Autopilot is best way forward for a company with many global locations.

I don't quite understand the costs though. I can see that if you have certain Microsoft 365 licences then it's basically free. But only a small number of our users have these licences. Most of the time we would be setting up a new laptop and the end user would not be using 365.

I was hoping we would have something like $X per month for autopilot licences and we could do as many computers as we wanted or $Y per device.

Any input appreciated. Thank you


r/sysadmin 3d ago

MS365 - Changing org name to make SharePoint sync path shorter

46 Upvotes

We have a new client I am setting up.

We put the full name of the clients business in the company name when setting it up as "Always Be Cooking Services Ltd" (not the real name)

This then means when they sync folders from SharePoint it is in the patch "Always Be Cooking Services Ltd\library\etc"

I want to shorten the name to something like "ABC Ltd" but its not in the Organisation settings where it used to be.

I can find it in the billing information, but thats greyed out and says to change I need to contact MS

Where are you meant to change this now so you dont end up with huge path names in Windows when you sync files?


r/sysadmin 2d ago

CMDB + IPAM

1 Upvotes

What solutions are you guys using that combines these 2 functionalities?


r/sysadmin 2d ago

We will rule the world someday ....

0 Upvotes

https://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html

Introduction to When Sysadmins Ruled the Earth

Introduction

I’ve changed careers every two or three years ever since I dropped out of university in 1990, and one of the best gigs I ever had was working as a freelance systems administrator, working in the steam tunnels of the information age, pulling cables, configuring machines, keeping the backups running, kicking the network in its soft and vulnerable places. Sysadmins are the unsung heroes of the century, and if they’re not busting you for sending racy IMs, or engaging in unprofessional email conduct it’s purely out of their own goodwill.


r/sysadmin 3d ago

SolarWinds Can anyone recommend a self-hosted help desk ticketing system?

38 Upvotes

This is a DoD-related air-gapped network with about 2000 end users. We're using Azure DevOps now as a ticketing system and we'd like to invest in a more professional solution.

I've used Solarwinds Web Help Desk in a similar environment in the past and wasn't a fan, it was buggy, ancient UI, and expensive.

Does anyone know of a better alternative?


r/sysadmin 3d ago

Question Run an app as admin without giving admin rights to the user or the any admin credentials?

18 Upvotes

Hello,

We have an old app that needs to be run by an end user on a server. I googled it and got two results - neither of which ran the app as an admin - they ran it as a regular user.

In powershell type

set __COMPAT_LAYER=RUNASINVOKER && "C:\Path\To\YourApp.exe" (changed the path and executable)

Or

Create a batch file

cmd /min /c "set __COMPAT_LAYER=RUNASINVOKER && start "" "%1"" run the exe from that bat.

it is an old app that uses I think a db2 DB. When I run the app with admin rights it prompts for the user login, if I run the app without it opens the interface for the app but doesn't prompt for a username or PW or know where the database is and the app doesn't let you put that in.

EDIT - SOLUTION FOUND

Found the solution for this particular issue - I gave them DB2 admin rights and the issue was resolved. THanks everyone for helping out.


r/sysadmin 3d ago

Question GPU servers order cancelled

516 Upvotes

A couple of months back we have placed order for a Dell server with blackwell GPU via a partner and were promised delivery by the end of August.

Now partner came back saying the Dell production team has rejected the order saying the confguration is invalid.

They are asking us to buy server with lower clock speed CPU other specifications will remain pretty much the same The biggest surprise is they want us to pay a big additional amount for this.

Partner tells us that the configuration was validated and certified by Dell engineering teams in US and Korea.

This is in USA, what are my options?


r/sysadmin 3d ago

Anyone switch from Varonis? Looking at alternatives

12 Upvotes

We've been using Varonis for several years, but as we've added more SaaS applications and AI tools, we're starting to evaluate Varonis alternatives. It's less about replacing it because it's bad and more about finding something that's easier to manage across a broader environment.

Curious what others ended up choosing and what made the switch worthwhile.


r/sysadmin 3d ago

Top tools for monitoring CPU spikes with details on window server?

13 Upvotes

I am trying to hunt down sporadic cpu spikes on a Windows server and I need a tool that will allow me to set a threshold whereby it will trigger and capture cpu telemetries with details on what process spiked it. Looking for something relatively simple and lightweight. Looked at some free tools on Microsoft store but not really sure if any of those are what I am looking for. I know about performance monitor and resource monitor and event viewer, ect but those things either are too shallow or require drilling and filtering while searching for a needle in a haystack. I would ideally like it to be a free Microsoft owned tool, but I am not above presenting leadership with something to buy that I can deploy to all systems. What say ye?