82
Python
4y

Had to wake four people up at 2 am to fix a crashing service.

10/10 would deploy to production on Friday night again.

Comments
  • 3
    I also deployed to production this night and I also broke it, but I managed to fix...most likely because I had to as I am the only one who knows this kind of stuff in our company
  • 2
  • 2
    A man with a middle finger would have been apt
  • 13
    Are you being paid for standby? No? Then fuck it and let it sit until Monday. That's the only way how employers learn not to do that shit.
  • 0
    @Fast-Nop ¯\_(ツ)_/¯
  • 7
    Unless it's a major release and the weekend is your buffer to get everything done before Monday morning,

    YOU DO NOT FUCKING DEPLOY ON FRIDAY!

    I hope that's easy to understand.
  • 0
    @C0D4 To my defence it was so late when I deployed it might actually have gone past 12 am, thus it was probably Saturday morning.
  • 5
    @Python it was Friday somewhere on the globe 😉
  • 2
    @C0D4 Haha, indeed. Now I have a reason not to deploy on Thursday’s as well 😉
  • 0
    Ok how did you find 4 whole DevOps people, I thought they were imaginary
  • 0
    We avoid deployment on Friday like Black Plague but sometimes it just can't be avoided.
  • 3
    I find it's kind of extreme to say you should never deploy on Fridays AT ALL. As long as it's early it's fine by me.

    If you don't dare to do friday morning deploys - you're basically saying you're scared it will take more than 6 hours to detect the issue and roll back or fix it. And if you have issues that take that long to be discovered you could basically not dare to deploy on a regular monday afternoon either - cause then the issue could be in prod for 10 hours before people start working again on tuesday.
  • 1
    Deploying to production on a Friday is happy and exciting. No risk no reward LOL
  • 0
    You uh... gonna do it again today?
  • 1
    @N00bPancakes Jokes on me, there’s a change freeze now so I can’t even if I wanted to.
  • 0
    And this is why my phone is off during the night! No PM is calling me at 2am! Haha
  • 0
    @C0D4 to be honest if you have a good infrastructure you could deploy at every hour of the day also on Friday and during the weekend.

    Saying don't deploy on Friday actually implies that you don't trust the infrastructure.🤷‍♂️
  • 1
    @Python are those code freezes working at your company! Nice!

    At my company PMs and other devs have been saying those words only thing is that there hasn't been a accual code freeze for years!
  • 1
    @darkvoid Optimism is rooted in lack of either experience or intelligence. Murphy's law is always just around the corner.
  • 2
    @darkvoid your right, you "could" deploy 24/7, but you obviously haven't had a release go bad and spent your weekend or following week for that matter trying to clean up the mess.

    Sometimes, a rollback is not an option.

    Bugs can and will always creep through, expected use cases vs required use cases don't always align.

    It's more then trusting your infrastructure, that I trust. it's being able to trust the entire process and the people involved from business representatives to BA, the dev and QA understanding what both those want and don't want, and that everything being pushed through meets expectations, not just requirements.

    But hey, if you trust everything to be right all the time, blindly forward you shall go.
  • 0
    @C0D4 I have had multiple production releases go bad and have spent time debugging production issues at night. But I agree it's more than trusting your infra.
  • 1
    @darkvoid Yeah, the freeze is enforced in the CD pipeline, so we just merge as usual but the changes just won’t get deployed to prod while the freeze is active. We have a core team that handles all that stuff, it’s really a luxury setup that I’ve missed dearly on my previous workplaces.
  • 0
    @AtuM Not sure if you're serious or sarcastic here but I choose to take your saying to heat here.

    Even if it was sarcasm I think your quote is amazing :D
    "That is how Fridays are done right.
    No fear, just sweat, good work and party time."

    Truly a rule to live by
  • 1
    @AtuM Sounds like a great time. 🍺

    Just curious: was there ever a case when you got interrupted by someone saying they found an bug in the latest release and you had to fix it?

    (I'll admit I had that happen to my team twice during the last 5 years. But it was never really an issue because it was issues we probably wouldn't have discovered by ourselves even if we had been at the office. And we could basically just roll-back with 1 click on a phone)
Add Comment