Jim Hirschauer: Hey everybody.
Welcome to Ship Talk, the SRE
edition.
I'm Jim Hirschauer, your host
for today.
Ship Talk is a DevOps podcast
brought to you by Harness, the
software delivery platform, and
the SRE edition focuses on
reliability topics.
My guest today is Brian Finster.
Thanks for joining us.
Brian, you're from Defense
Unicorns and welcome to the
show.
Bryan Finster: Yeah, thanks so
much for having me, Jim.
Jim Hirschauer: Brian, why don't
you take a minute to share your
background.
Tell us a little bit about
defense unicorns, and talk about
any other projects you might
wanna fill us in on.
Bryan Finster: Sure.
So I've been a software engineer
for nearly 30 years now.
I, I still do that most of the
time in supply chain software,
which really kind of informs my
attitudes towards delivering
software because it's, you know,
it's, it's a supply chain
problem, just a different kind
of thing.
We're, we're moving for the last
several years I've been working
on how do we improve the flow of
software delivery, focusing on
implementing continuous delivery
as a workflow.
You know, what are the
challenges around that?
Yeah.
And you know, I spent a long
time at Walmart doing that, and
then I, I got stolen away by
defense unicorns and.
Defense unicorns, you know,
we're trying to solve some
really hard software supply
chain problems.
You know, how do you deliver you
know, a hardened secure solution
to top secret environments, you
know, that are air gap where you
can't, you don't have access to
normal tools, that sort of
thing.
Yeah.
You know, just really trying to
solve.
That, that problem space, if you
can solve that delivery problem,
everything else is easy.
Jim Hirschauer: Yeah, that's a
very difficult problem for sure.
Bryan Finster: But that's, I
love working really hard
problems and this is a real fun
one.
Jim Hirschauer: Great.
Sounds exciting.
All right, so look, I appreciate
the background.
I feel like you're gonna drop
some knowledge on us today.
I'll try.
Before we get there, though
yeah.
On this podcast, we'd like to
have a little bit of fun first.
And so what I'd love to do is to
hear about your hobbies.
Everybody has hobbies outside of
work.
And why don't you pick one of
your favorite hobbies and, and
fill us in.
Bryan Finster: Well,
motorcycling I think is, is one
of these things that I really
geek out about.
I mean, I'm like a, you're not
gonna catch me saying this brand
is the best, cuz I'm like a nerd
about motorcycles, new bikes
come out and I'll go and try to
test ride'em.
Not because I wanna buy one just
cause like, Hey, what's this new
bike like right now?
Nice.
Yeah.
Love motorcycles.
And I, I love distance riding.
I, I, I ride a BMW GS adventure.
And I've taken that thing on
thousand mile trips in a day.
You know, it's, it's fun.
It still kind of gets the
adrenaline junkie part of me
going as well.
I love diving into corners.
Jim Hirschauer: Nice.
Yeah.
Well, I don't know a whole lot
about motorcycles.
I definitely know that BMW makes
some fantastic motorcycles for
sure.
And it sounds like you have an
adventure bike.
Those are the ones that you can
kind of take on road offroad,
right?
Bryan Finster: yes.
But I'm realistic.
They make bikes that are
slightly more reliable than
Harley Davidson's and much less
reliable than Honda's.
You just need to know where
you're sitting.
Okay.
In a reliability space.
Yeah.
Jim Hirschauer: Good to know.
So any, any fun stories to share
about riding your motorcycle on
these long trips?
Bryan Finster: Just, you know, I
love getting there and I love
going places where there, where
I can be challenged.
You know, there's a road in
north North Carolina, a
Tennessee border called the
Dragon Us 1 29.
I've heard of it.
Yeah.
318 corners and 11 miles.
Yeah.
And taking a big adventure bike
like that, and a big guy like
me, I'm six six, I'm not a small
person.
And diving into those corners
and, and just challenging how do
I do this corner better?
How do I do this corner better?
I mean, it's a lot like how I
try to deliver software.
It's like, you know, small
incremental improvement all the
time.
You know.
Jim Hirschauer: Awesome,
awesome.
Diving into corners.
Bryan Finster: Yeah.
You know, and, this is something
that I sometimes try to use this
as an analogy because there's
habits that you have to develop.
Motorcycles are very
counterintuitive.
In a car, sometimes the safest
thing to do is to hit the
brakes.
You're coming into a corner too
fast in a car, you, you wanna
slow down and reduce your energy
coming into that corner so you
don't run off the corner.
The problem is, is that if you
apply habits you learned in a
car to a motorcycle, it doesn't
work the same way.
You need to understand the
physics of the motorcycle.
I mean, motorcycles turn by
leaning.
Yeah, right?
Yep.
And if you hit the brakes on a
motorcycle, the bike stops
leaning, it stands up.
And so if you're coming into a
corner too fast in a motorcycle,
you need to just be smooth and
just lean harder.
And go through the corner
because the most dangerous thing
you can do is just slam on the
brakes, stand the bike up, and
they just run right off the
corner.
And this is a lot like the
behaviors I see when people are
trying to transition to
continuous delivery workflows
is, is that they, you know,
they'll hit a bump along the way
and they'll start slowing down
delivery, which means that the
delivered batches get bigger.
But the safety from CD comes
from delivering smaller batches
and getting faster feedback and
it, it's the same sort of
mindset.
It's counterintuitive, but you
need to.
Deliver more frequently in
smaller chunks rather than slow
everything down and panic
because you hit a roadblock.
Jim Hirschauer: Yeah, I love
that analogy.
So, leaning into your, your
software delivery practices,
that's a great segue into the
main topic for today's show.
You know, the SRE edition of
Ship Talk, we like to focus on
reliability and resiliency.
Right now we're having some
conversations industry wide,
about efficiency and on the show
we did last week, I spoke to
Matt Schillerstrom about
efficiency and I wanna have that
similar conversation with you
today and maybe expand into the,
the software delivery portions
of this.
So, you know, Overall efficiency
is a really hot topic right now,
and especially given the current
economic environment and as
companies, yeah, yeah.
As companies are looking for
different ways to reduce their
costs and increase productivity,
automation, reliability,
resiliency.
These are becoming top
priorities.
So I'd love to hear from you,
what's your take on how
companies can be successful
given the fact that efficiency
and reliability, they're often
working against each.
Bryan Finster: Well, I think
that a mistake a lot of
companies make is they say,
we're gonna become more
efficient so that we can be more
productive.
And by that they mean generate
more output.
Mm-hmm.
But you know, the perspective I
have, like I said, I've been a
developer for a long, long time,
and except for a small well,
It's been a long career except
for a portion of time when
operational responsibility was
stripped away from me.
Where for, to improve developer
productivity, make support of my
application, go to another team.
I have had operational
responsibility for what I build.
Right, and I don't mean I own
the infrastructure and all the
tools and everything.
I mean that I'm building the
application and if it breaks,
I'm the one getting woken up in
the middle of the night to fix
the application.
And ops is the reason for all of
this.
We want to become more efficient
about how we deliver software.
so that we can, you know, make
smaller batch sizes of of work
so that we can fix production
faster.
Right.
So the smaller batches of work
allow us to uncover the pain
points preventing us from
delivering high quality change.
Right?
Right.
It could be processed, it could
be how we test whatever, but
shoving down those batch sizes,
uncovers all that pain.
But to do that, we have to.
Anytime we find waste in our
flow of delivery, we need to
remove that waste handoffs to
other teams and lack of access
to product information we need,
whatever that is, how do we
improve the supply chain of
communication and optimize our
processes so that we can keep
shrinking that change set size
down so that we can every single
day build the muscle memory.
Of fixing production.
I mean, this is the thing about
cd.
It's not about speed.
It's about building that muscle
memory.
We are delivering daily.
Yeah, hopefully multiple times a
day if we're on a high
performing or a team, so that
when we wake up at three o'clock
in the morning, we're Oh, and,
and I left this out, and we're
only ever using our emergency
flow.
We're using our hot fix process
to deliver every single change
because we are validating that
our hot fix process works very,
very well.
We're stressing that system of
delivery to uncover.
Broke breaks during the normal
business hours, so we don't hit
those breaks in the middle of
the night when something breaks
because something will break
when we're asleep.
Yeah, right.
And I focus on CD because I
wanna go to sleep at 3 0 5 when
I woke up at three o'clock with
a system that's down.
Right.
I've spent too many years on
pager.
It was too many sleepless nights
not to do continuous delivery.
Jim Hirschauer: So you said
something really interesting.
You said that it's about ops.
It's about being able to recover
your systems as quickly as
possible.
Yes.
But what does that mean for Dev?
Because at the end of the day,
you also said in your time in
Dev, you were responsible for
what you deployed.
Right.
That software that you deploy.
Bryan Finster: I am.
I am still a dev.
Yeah.
And here's just the hard truth.
I was talking to Andrew Clay
Shafer about this yesterday.
This whole thing about coddling
developers to take away
responsibility for the things
that they build so they don't
feel the impact of their
decisions drives poor quality.
If an organization wants high
quality software, they will give
ownership to teams.
And ownership means taking
responsibility for the, your
architectural and, and tech
stack decisions.
Mm-hmm.
you don't wind up with a team
voluntarily building something
with five different languages if
they also have to fix it in the
middle of the night.
Right.
They're gonna optimize for clean
code, clean architecture, easy
to understand.
Simple and effective.
Right?
Right.
They're gonna optimize for not
being woken up and that's what
drives quality.
And when you separate those two
things, you kill that quality
feedback loop.
And developers don't know that
they're breaking stuff and a
some will just start playing.
But the, I'm a developer, I
don't, I have no problem holding
my peers accountable for being
professionals and taking
responsibility for the work that
we do.
Jim Hirschauer: And how does
that translate into the software
delivery life cycle as a whole?
You mentioned continuous
delivery, but when we talk about
like reliability and resiliency,
that shouldn't start in
production, right?
By the time you get to
production.
It's there, right?
And so whatever's there is
there, and it's gonna be as
reliable as you've made it.
And whatever users are gonna do
to it, they'll do.
So where does it start and how
do you make that part of your
software delivery life cycle?
Bryan Finster: Well, it starts
with your pipeline.
The, the purpose of a CD
pipeline is not to deliver
software.
The purpose of a CD pipeline is
to prevent bad software from
being delivered.
Mm-hmm.
And so we need to identify what,
number one, what is our
definition of deliverable.
Can we even describe it?
If we can't, we already have a
problem If we need to have a
definition of deliverable.
It's a certain level of
security, a certain level of
performance for what it is we're
delivering on.
You know, this particular use
the problem we're trying to
solve, and then codify that in
the pipeline.
And then we start shipping down
the pipeline to find out how
wrong we are about defining our
definitions in the pipeline and
hardening our pipeline.
When I, when I've worked with
teams in the past on this, I
tell'em that, look, you're a
product team, you own this
business problem and solving
this business problem, but your
primary product is the pipeline
delivering it.
Your job is to, whenever
something breaks, because it
will, your response is, Where do
we put something in the pipeline
to quickly make that not happen
again?
You know, and this comes to, we
have to design efficient tests
that are capable of giving us a
signal when things broke, about
where it broke so we can fix it
quickly before it went to
production.
And when things break in
production, putting a another
test in place that will keep
efficiently doing that, we need
to keep track of how long that
takes to run that pipeline.
Because if it takes you four
hours to run your pipeline,
you're not going to, when
there's an emergency you're
gonna come up with some hot fix
process that's not well tested
and pour gasoline on a dumpster
fire.
I know I've done it, right?
Yeah.
So, yeah.
You know, and, and so it's,
you're constantly going, okay,
we're gonna ship, we're gonna
find out what breaks, we're
gonna just continue improving
the things that break.
Jim Hirschauer: When you put all
that together, it sounded like
you were saying our pipelines
are not just about delivering
new features.
Right.
They're there to make sure that
we're delivering quality
software to our end users.
Bryan Finster: They're our
safety net.
Yeah.
Yeah.
And, and we are, we own our
safety net.
We will make mistakes.
The automation is there to help
us.
Avoid making them twice.
Jim Hirschauer: So, from that
perspective, it sounds like
you're saying that resiliency
engineering should be part of
this overall process, and not
only in production, but baked
into your software delivery
pipelines as well.
Bryan Finster: Everything.
If you're starting a new team,
working on a new product, Or
even if you're just starting a
new service for a product that
exists.
Mm-hmm.
right?
Feature zero is your pipeline.
Yeah.
If you don't have a way to get
to production, to invalidate
your assumptions about what
you're building, everything else
you do is waste, and you're just
piling defect on top of defect
without knowing it.
Jim Hirschauer: You know, it's
interesting.
We were just working with a
company out in Australia that
Yeah.
Was releasing a new product,
right?
This is an established company,
but they're releasing new
product and their take on
resiliency engineering was that
they needed to get it
implemented before they released
their new product.
And that was honestly for me,
that was the first time I had
ever heard a company make that
statement.
Their opinion was if we launched
this new product, and it
provides a poor customer
experience.
It's worse than not launching
the product at all for them.
They didn't want to to create
that bad first impression.
Bryan Finster: I totally agree.
Yeah.
But you also need to figure out
how I can get this into
production to validate some of
our assumptions about it running
in a production environment
without also exposing it to
people.
In a way that's not good.
Right?
I mean, you want, you, you, you,
you want to deploy.
There's, there's many, many
times when I'm deploying the
production things that nobody,
but I know, I mean, it's, it's
being tested by the fact that
it's not blowing up and dying in
that production environment.
One of the things I work on is
I'm a contributor to minimum
cd.org and we have just a list
of problems to solve.
Right.
It's not implementation do CD
this way.
It's, here's problems to solve
that if you solve these
problems, now you're doing
continuous delivery.
Right.
One of those problems is
production-like test
environments, but it's
production-like.
Yeah.
the odds of you having a test
environment that matches
production are zero.
Yeah.
Absolutely right.
Yeah.
And so, you know, getting
something out there in a way
where you can do some validation
that it will actually run and
doesn't break anything without
hurting anything while you do
it, is critical.
Jim Hirschauer: So is this
validation, is this strictly the
realm of chaos engineering type
of solutions?
Or is there more than that that
goes into it?
Bryan Finster: I mean, it's far
more than that.
Not only is it resilient, but
does it perform it, is it
functional?
Right?
I mean, what tests can we throw
at it in production?
Yeah.
To validate its does in
production what it should do.
I mean, we, there's all sorts of
mistakes we can make.
We could have configurations set
incorrectly.
We only find out when we
deliver.
Jim Hirschauer: Yeah.
Are you, so you're advocating
testing and production?
Bryan Finster: Absolutely.
But testing production doesn't,
doesn't mean we only test in
production.
It means we also test in
production.
I, I think people really need to
understand that all of this is
an integrated system.
You know, there's this.
People see these different words
being used.
They'll hear agile and DevOps
and continuous delivery, and
that they'll think that these
are all different things and
they're not.
What we're trying to do is we're
trying to implement a system of
delivery that gives us.
Rapid feedback from idea all the
way to delivery, right?
Every single step in between
rapid feedback that we have a
problem.
Jim Hirschauer: Brian, I love
your, viewpoint on this holistic
way of thinking about software
delivery because in reality,
this is what our end users, it's
what they experience.
At the end of the day,
everything that led up to
delivering that software to an
end user, it is what their
customer experience is.
Bryan Finster: You know, and,
and, and by the way support
documentation, all of those are
part of, those are all features.
Those are all part of the
product.
Make sure they're good.
Jim Hirschauer: Absolutely.
Yeah, I couldn't agree more.
There's so many times where I've
personally been looking at docs
to try and figure out how do I
use this thing, right?
And what I'm looking for isn't
documented All right, great.
So listen, let's take a quick
transition here because on this
podcast, we also do another fun
segment at the end.
So, one of the things I love to
ask and, and I found that people
really do like talking about,
because at the end of the day,
we're all humans and we, we mess
up.
And when we work in IT,
sometimes we have some pretty
interesting mess up.
So, Brian, I'd love to know,
what's your worst it mess up?
Bryan Finster: Well, I had to
write down a list because I've
been doing this for a while and
I've, I've messed up right.
And this, this particular mess
up is one of the reasons why I
say that nobody should ever have
access to production at all,
except through a pipeline.
Mm-hmm.
Okay.
Okay.
Because I was on a production
system in the wrong path.
I was in bin.
Yeah, right.
I mistyped a command cause I was
trying to clear a log file, oh.
And I was like, why is it taking
so long to delete that log file?
Jim Hirschauer: Did, did you
delete your whole operating
system
Bryan Finster: Well it was, it
was the primary application that
consisted about 800 different
discreet programs.
Right.
So it was the, the warehouse
management system?
Yeah.
Right.
So I was deleting the warehouse
management system.
Oh, wow.
From that, from that box, it was
out on the edge.
Now I rapidly went to another
box, you know, another DC and
inserted FTPing applications
across and hoping that we had no
version.
We were gonna kill and then
crossed my fingers and then just
didn't tell anybody.
Never never touch production
Jim Hirschauer: So did it work
out?
Did you get away with it?
Bryan Finster: I got away with
it, yeah.
You know, there was another time
when I didn't get away with it,
but my tech lead covered for me
where I, I deleted an
application that only existed in
one distribution center that
calculated tax for Argentina.
So for 30 days, Argentina didn't
have any tax calculation going
on, and that was, yeah.
Oh my.
Never touch production.
Jim Hirschauer: Yeah, I agree.
So I used to be a systems
administrator a long time ago,
and we worked in those
production systems all the time
and it, it made me nervous even
though I was really comfortable
in there and I had, you know,
multiple factor authentication
to get the root user on Unix.
Wow.
Did you have to be careful in
there?
Bryan Finster: My office mate at
a previous company he was tech
lead for public stores and he
thought he was deleting the
inventory table from a
development system.
He deleted a production
inventory table and from that
point on, I always had a
different color screen on my
terminal emulator for production
versus development.
Jim Hirschauer: Oh, that is a
brilliant.
Absolutely.
So, yeah.
Lesson for everyone listening.
Yeah.
If you have any access to
production systems and
non-production systems, like
Triple Verify which one you're
on before running that command.
Especially if, if it's a
destructive command.
Bryan Finster: Absolutely visual
verification somewhere PROD, you
know, giant letters or screen
colored something cuz you will
make a mistake.
Jim Hirschauer: Absolutely.
All right.
Well, Brian it's been wonderful
talking to you.
Thank you so much for being our
guest on the show today.
Again, love your idea of looking
at software delivery from a
holistic perspective.
Completely agree with it.
Thank you for sharing your
humanity with us and, and, you
know, the, the mess up that
you've had.
We've all done it.
So yeah.
We're all, we're all in the same
boat with you.
Yeah, yeah, for sure.
And to everyone listening if
you're interested in being a
guest on Ship Talk, if you're an
SRE or if you're in a DevOps
related role, Just feel free to
send us an email.
Send that email to podcast at
ship talk.io and we'll get back
to you.
That's all for now.
Until next time.
Bryan Finster: Thanks, Jim.
Jim Hirschauer: Thanks Brian.