GOLFMK8
GOLFMK7
GOLFMK6
GOLFMKV

CSB3: Corprin And Knotical Immediately Sus For Causing Boat Accident

jay745

What Would Glenn Danzig Do
A couple weeks ago, I ranted about one of our teams causing an issue by testing stuff in a prod environment.

Well, that came back up last night and blew the fuck up.

Yesterday afternoon, someone was looking at one of the reports we put together, and went "Hey, this little piece here? This isn't right." So we look at the system of record (that had tests done in prod), and it matches what's on our report. It's correct. So we peel back another layer, and look at the data that feeds our system of record.

And sure enough, he's correct. It's wrong. Our system of record has data inaccuracies in it.

Fuck.

The reason that's a big deal, is because those reports have already been distributed and are about to be presented in 2 hours to a bunch of regulators, the boards of directors, and just overall a pile of people making 7 and 8 figures. And while the actual inaccuracies themselves are very minor, it calls into question the data and legitimacy of everything else in the report.
I thought you guys had a risk manager to make sure this kind of stuff doesn't happen?
 

sterkrazzy

Autocross Champion
A couple weeks ago, I ranted about one of our teams causing an issue by testing stuff in a prod environment.

Well, that came back up last night and blew the fuck up.

Yesterday afternoon, someone was looking at one of the reports we put together, and went "Hey, this little piece here? This isn't right." So we look at the system of record (that had tests done in prod), and it matches what's on our report. It's correct. So we peel back another layer, and look at the data that feeds our system of record.

And sure enough, he's correct. It's wrong. Our system of record has data inaccuracies in it.

Fuck.

The reason that's a big deal, is because those reports have already been distributed and are about to be presented in 2 hours to a bunch of regulators, the boards of directors, and just overall a pile of people making 7 and 8 figures. And while the actual inaccuracies themselves are very minor, it calls into question the data and legitimacy of everything else in the report.
F
 

Acadia18

Autocross Champion
What level of blame are you involved in this?

Zero. We report off what's in the system of record (SOR). That's the entire point of having a SOR. Something regulators and government agencies can have access and be directed to to ensure we're doing our job as a company. So we take the data from that, and turn it into nice looking reports. The systems team who maintains the SOR are the ones who fucked up. That's the good thing at least, is it's not just a bunch of finger pointing. The systems has already weighed in, explained what happened and how it got fucked up, and advised what technical redundancies will be taken in the future to ensure it doesn't happen again. But the easiest way, is just don't fucking test shit in prod! "We ran a data feed on 2/13 to test for the new fields added". Why the fuck would you do that?!?

I thought you guys had a risk manager to make sure this kind of stuff doesn't happen?

We have 30 levels of checks and balances. On the analytics team, we can't know every miniscule detail of every piece of information in the SOR. That's why there's actual teams upon teams who own the actual data that we report on. We pull data out of the SOR, have a high level understanding of everything, and turn it from a CSV with hundreds of columns and thousands of rows into something digestible. And that's pretty much what happened. A risk manager was looking at the report, and said "Hey, you guys are saying this issue was late receiving an update. It wasn't, it received an update before month end on 2/27". Well, not according to the SOR it didn't. But that's the kind of thing that's their job to know. It just sucks it was pointed out in the 11th hour.

tl:dr; DON'T TEST SHIT IN PROD
 

jay745

What Would Glenn Danzig Do
We have 30 levels of checks and balances.
Sounds like you need 31

















..













.







29gymp.jpg
 

riceburner

Autocross Champion
Thats good bud, not your direct problem lmao.

But sounds like you got some stuff to do today then huh
 

Acadia18

Autocross Champion
Thats good bud, not your direct problem lmao.

But sounds like you got some stuff to do today then huh

Kick back, watch this week's Ted Lasso, maybe install my RacingLine coolant hose :ROFLMAO:

At this point, everything is pretty much done. Was on meeting till like 6:30 last night about how the issue was going to be addressed and what to do. Spent further time plowing through logs to make sure no other updates were missed and all other data is accurate. I'll give it another look this morning just to be safe, but at this point pretty much everything affected has been identified, it'll be spoken to at the committee this morning, and it is what it is.
 

Acadia18

Autocross Champion
Everyone has a test environment, it's just a smaller portion of the population who's got a separate production environment

:ROFLMAO:

We have a test, QA, and prod environment. There's zero reason the systems team should have done what they did.
 

riceburner

Autocross Champion
1679577827237.png
 

riceburner

Autocross Champion
Whenever this brother posts a new 'cast I'm in on it!
 

Nineeightyone

Autocross Champion
I don't get to touch the various environments anymore at work, I now purely work on hardware. But at home? Hooooo baby. Aside from machine image backups, I play it fast and loose -- the upgrade from Server 2016 to Server 2019 was pretty sketch, and while the rollup from ESXi 6.7 to 7 didn't go too poorly, the next one was a shitshow as I panic tried to roll back remotely after learning my evaluation license would die after a short period. No idea if they've changed that, but I have no plans to pay for it in the future, if nothing else I'll move to Hyper-V.
 
Top