So I’m working on a server from home.

I do a cat /sys/class/net/eth0/operstate and it says unknown despite the interface being obviously up, since I’m SSH’ing into the box.

I try to explicitely set the interface up to force the status to say up with ip link set eth0 up. No joy, still unknown.

Hmm… maybe I should bring it down and back up.

So I do ip link set eth0 down and… I drive 15 miles to work to do the corresponding ip link set eth0 up

50 years using Unix and I’m still doing this… 😥

  • twinnie@feddit.uk
    62·
    4 days ago

    I knew a guy who did this and had to fly to Germany to fix it because he didn’t want to admit what he’d done.

  • iriyan@lemmy.ml
    6·
    3 days ago

    I still prefer net-tools and use ifconfig eth0 up That ip mess I’d rather do without, and those funky UU device/interface names I wish them out of my system

    By the way, what system/init/svc manager are you using? With 50y in your back, cron job to check if it is up and resetting it while you are away. You can always remotely cancel the cronjob … but it will be a new mistake not the old one :)

    I started on Irix and ultrix if you remember those, what would I know :)

  • InnerScientist@lemmy.world
    31·
    4 days ago

    I have a failsafe service for one of my servers, it pings the router and if it hasn’t reached it once for an entire hour then it will reboot the server.

    This won’t save me from all mistakes but it will prevent firewall, link state, routing and a few other issues when I’m not present.

  • moonpiedumplings@programming.devEnglish
    6·
    3 days ago

    Use cockpit by Red Hat. It gives you a GUI to make networking changes*, and will check if the connection still works before making the change. If the connection doesn’t work (like the ip addresses changed), it will undo the change and then warn you. You can then either force the change through or leave it be.

    *via NetworkManager only.

    • caseyweederman@lemmy.ca
      2·
      3 days ago

      That’s probably because of netplan, right? You should be able to get the same results with just netplan try.

      • moonpiedumplings@programming.devEnglish
        2·
        3 days ago

        Netplan is an abstraction layer, so it can go over systemd-networkd, NetworkManager, or iproute. I suppose it’s better though, because it can be used with multiple backends.

              • moonpiedumplings@programming.dev
                2·
                3 days ago

                No. Netplan uses it’s own yaml format, which people would have to learn and use. I don’t want to do that, I would rather just configure my existing networkmanager setup, rather than learning another abstraction layer over what is already an abstraction layer.

                I understand that cockpit (and similar type tools) are “the whole kitchen sink” of utilities, and it may seem like they come with more than you may need. But that doesn’t change the fact that they get the job done, and in some usecases, are better than dedicated tools.

    • catloaf@lemm.eeEnglish
      315·
      4 days ago

      Or use some kind of molly guard. Or have an OOB management channel.

      You’d think you’d learn from your mistakes after one or two of them, not fifty years’ worth…

        • DasFaultier@sh.itjust.works
          20·
          4 days ago

          after hours

          I’ve configured PAM to not let me login remotely after hours, because I just know that someday I’ll want to fix “just this tiny thing” and I’ll break production because I’m too tired. I clearly need protection from myself, and this is one slice in Dr.Reasons’s Swiss cheese model.

          Don’t let the people drag you down, this happens to all of us.

      • IsoKiero@sopuli.xyzEnglish
        8·
        4 days ago

        You’d think you’d learn from your mistakes

        Yes, that what you’d think. And then you’ll sit with a blank terminal once again when you did some trivial mistake yet again.

        A friend of mine developed a habit (working on a decent sized ISP 20+ years ago) to set up a scheduled reboot for everything in 30 minutes no matter what you’re going to do. The hardware back then (I think it was mostly cisco) had a ‘running conrfig’ and ‘stored config’ which were two separate instances. Log in, set up scheduled reboot, do whatever you’re planning to do and if you mess up and lock yourself out the system will restore to previous config in a while and then you can avoid the previous mistake. Rinse and repeat.

        And, personally, I think that’s the one of the best ways to differentiate actual professionals from ‘move fast and break things’ group. Once you’ve locked yourself out of the system literally half way across the globe too many times you’ll eventually learn to think about the next step and failovers. I’m not that much of a network guy, but I have shot myself in the foot enough that whenever there’s dd, mkfs or something similar on the root shell I automatically pause for a second to confirm the command before hitting enter.

        And while you gain experience you also know how to avoid the pitfalls, the more important part (at least for myself) is to think ahead. The constant mindset of thinking about processes, connectivity, what you can actually do if you fuck up and so on becomes a part of your workflow. Accidents will happen, no matter how much experience you have. The really good admins just know that something will go wrong at some point in the process and build stuff to guarantee that when you fuck things up you still have availability to fix it instead of calling someone 6 timezones away in the middle of the night to clean up your mess.

  • dependencyinjection@discuss.tchncs.de
    5·
    3 days ago

    Not SysAdmin but about a year into my first software engineer job I was working on the live DB in SQL without using BEGIN TRAN ROLLBACK TRAN.

    Suffice to say I broke the whole system my making an UPDATE without a WHERE clause. Luckily we have regular backups but it was a lot of debugging with the boss before I realised it was me.

  • apt_install_coffee@lemmy.ml
    7·
    3 days ago

    A few months ago I accidentally dd’d ~3GiB to the beginning of one of the drives in a 4 drive array… That was fun to rebuild.

    • IngeniousRocks (They/She) @lemmy.dbzer0.com
      3·
      3 days ago

      Like 3 weeks ago on my (testing) server I accidentally DD’d a Linux ISO to the first drive in my storage array (I had some kind of jank manual “LVM” bullshit I set up with odd mountpoints to act as a NAS, do not recommend), no Timeshift, no Btrfs snapshot. It gave me the kick in the pants I needed to stop trying to use a macbook air with 6 external hard drives as a server though. Also gave me the kick in the pants I needed to stop using volatile naming conventions in my fstab.

  • Float@startrek.websiteEnglish
    16·
    4 days ago

    Every network engineer must lock themselves out of a node at some point, it is a rite of passage.

  • toynbee@lemmy.world
    7·
    3 days ago

    A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don’t recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn’t being enforced globally.

    Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or setenforce and set the mode to … Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn’t realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn’t login to the systems and suddenly understood the problem.

    Fortunately, it’s necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running … So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).

    When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.

    edit: While I could have done without the assault on my blood pressure at the time, it was an effective demonstration of our lack of readiness for enforcing mode.

  • sleepmode@lemmy.worldEnglish
    9·
    4 days ago

    I was on-call and half awake when I got paged about a cache server’s memcached being down for the third time that night. They’d all start to go down like dominoes if you weren’t fast enough at restarting the service, which could overwhelm the database and messaging tiers (baaaaad news to say the least). Two more had their daemon shit the bed while I was examining it. Often it was best to just lick it on all of them to rebalance things. It was… not a great design.

    So I wrote a quick loop to ssh in and restart the service on each box in the tier to refresh them all just in case and hopefully stop the incessant pages. Well. In my bleary eyed state I set reboot in the variable instead of restart. Took out the whole cache tier (50+) and the web site. First and only time I did that but that definitely woke me up.