S3, Ep2: Flynn from Buoyant & Cloud Native Happenings

03:52.57

Flynn

You know there's still an attitude among some of the software engineering folks that the ops side is somehow the lesser brethren of the software engineers and I think that that was even more prevalent back when it was Sysadmin instead of ops. Um, but.

04:02.89

Dave Sudia

Sure.

04:10.78

Flynn

There was I think the biggest shift was ah for me at that time was probably getting to a place where I wasn't touching hardware or wires or anything like that and suddenly I was one of the folks who was responsible for deciding. How the wires and such should be connected and what they were going to do and things like that and so it it felt like a really big change. Um I've heard it said that. The really great sysadmins are all programmers as well and the really great programmers have all spent time as sysadmins I Think that's very much true I Think the the perspective about how things go on the other side of the wall is is still really really valuable I think there's a lot of value in that and.

04:49.50

Dave Sudia

Sure.

04:59.24

Dave Sudia

Yeah, totally Well I mean because the point of devops was to break down the wall but let's be honest in any significant any you know large enough organization. No one can know everything and so it it really just becomes about improving the communication.

05:01.32

Flynn

We didn't talk about it much back then but it was real.

05:13.59

Flynn

Yeah.

05:19.30

Dave Sudia

You know and and trying to have the wall be made a glass but but but but having the experience on both sides you know, yeah helps massively.

05:19.87

Flynn

There's ah I often. yeah yeah I often have described devops as a concept as the radical idea that. The devs and the ops folks should actually trust each other to be. You know both working in the best interest of everybody and and to get each other's you know to get their jobs done. Um, that idea of Dev and ops actually trusting. There are counterparts on the other side of the wall. No matter what the wall is made of I still feel like that's something we got to work on more than you know I don't really feel like that's a solved problem yet.

05:58.37

Dave Sudia

Um, yeah.

06:01.99

Dave Sudia

No, it's I mean it's a human problem. So I don't know that it'll ever be solved. But.

06:08.12

Flynn

I I doubt it I think we could do things to ah I would like to hope that we can. Yeah yeah and I would like to hope that we can do things with you know, providing better perspective so that it's.

06:14.10

Dave Sudia

That's not to say it's not worth the effort to make it better is what? yeah yeah.

06:23.27

Flynn

Easier for the developers to understand that hey the ops folks have a real job and it's very very difficult and you know you need to you need to respect the fact that they are doing something really difficult and holding it all together and you know likewise the ops folks should.

06:27.76

Dave Sudia

Yeah.

06:40.22

Flynn

Probably understand that the software engineers have a real job too. Yeah.

06:43.31

Dave Sudia

Yeah, for sure I mean I yeah I spent 2 hours yesterday trying to figure out route 53 hosted zones. So I'm not going to like yeah ops is hard. Yeah, ah yeah, and so and so is making an app that works.

06:46.52

Flynn

Oh god opposite hard. Yeah I I feel for you and I'm sorry you had to whip burn those 2 hours that you're never gonna get back? Yeah yeah.

06:58.98

Dave Sudia

So Yes I agree with you that it's It's on both sides. Um, yeah, you know and and heck I'm I remember doing real ops here I'm just trying to like promote things. So Ah so that and on that note so as a more recent transition into from from software engineering into Tech evangelism. What what has that been like for you.

07:20.86

Flynn

That's a little um I don't feel like that was as abrupt a transition as people always assume because towards especially towards the tail end of my tenure at Ambassador Labs I was doing some of the evangelism Anyway, right.

07:25.63

Dave Sudia

Sure.

07:34.90

Dave Sudia

Sure.

07:38.20

Flynn

Um, but it's it's funny I think the most fascinating thing about that transition for me has been that I I have to remind myself every so often that I am no longer part of the engineering organization and.

07:50.95

Dave Sudia

Indeed.

07:54.25

Flynn

On the 1 hand this is really nice right? My life is no longer ruled by sprints and I am no longer on call which is oh dear god that is a beautiful beautiful thing. Um.

07:56.22

Dave Sudia

You are no longer on call.

08:04.51

Dave Sudia

I was talking to a colleague recently at another old old ex colleague and we're still friends and she was telling me I could be making so much more money if I did x or y or z and I said yes, but the most pressing thing I have to do right now is write blog posts and. And you had to put off this dinner for three weeks because you've been on call.

08:24.51

Flynn

Right? right? Um, lest anybody get the wrong idea that is not Dave and I implying that writing blog posts is easy. It turns out that wrangling english is difficult. Yeah.

08:38.25

Dave Sudia

No, it turns out that that's the board. That's a part of this job I'm the worst at ah, it was the most pressing thing it was still yeah.

08:42.34

Flynn

Yeah, you know? yeah, it's it could be tough right? I mean the the stuff that I've done recently has mostly been writing in a lot of english I put together a demo that I had to do earlier today. But honestly Regina Scott from one of the argo maintainers did. An enormous amount of the work on that demo which was beautiful and even at that going through in doing the demo There's a lot of points when you're putting the demo together. You're kind of like okay wait a minute. How are we going to talk about this It's not how do we get the demo to work. It's how do we explain? what's going on.

09:12.69

Dave Sudia

Yeah.

09:19.50

Flynn

This um this you know this business of Tech evangelism That's figuring out how things work and then figuring out how to explain it to people. Yeah, that's a challenge. Yeah.

09:30.60

Dave Sudia

It's teaching no and that's where I feel like coming into this role you know because I got hired ah into this role based off of my ah teacher or talking at Kubecon a lot and other conferences and stuff and I think that that's the the thing I've learned is that I was hired for talking and then I was asked to write and then I learned my limitations. Ah, because I think by talking and so that parsel and I and I and I have a background in teaching and so I do love teaching people by talking at them. Ah so I Oh yeah for sure.

09:48.31

Flynn

Ah, oh that's brutal.

09:55.46

Flynn

Teaching is a lot of fun when you're dealing with people who want to learn things and I've I've generally had the luxury that when I am teaching it's teaching people who want to be there and want to learn which is really nice actually.

10:10.62

Dave Sudia

Yeah, Okay, so here's another question for you on that? ah topic is you know? So we're both at places that have open source projects that have been donated cncf we both and then you know our companies both offer. Ah. Paid products on top of that. How do you balance that? How do you balance the the open source versus the paid and ah and you know I think people some people are more touchy about it than others and you know, but but at the end of the day we're all looking for solutions and we all know that you know there are reasons to to buy Software. So What's your take on that.

10:46.11

Flynn

My take it that is that there are reasons to buy software and people should pay attention to those reasons and they should buy it when it makes sense. Um, and obviously I'm snide's probably not the right word there but that's the the very.

10:50.67

Dave Sudia

Um, sure.

11:02.53

Flynn

Most biased way I can present that um I think that open source Software has a much better track record than close source software in basically everything that I think is important.

11:16.83

Dave Sudia

Sure.

11:19.30

Flynn

Particularly in terms of security and reliability I think open source software works better than the closed source model. Um, now. That being said, Ah, what was that line from serenity powerful need to eat sometime this month you know bills to pay got a kid. A family got a mortgage. We have a dog you know and the reality is that.

11:44.12

Dave Sudia

My dog has 3 kinds of allergies that require several expensive shots. So I feel you there? yeah.

11:50.20

Flynn

Oh ouch my dog thankfully does not seem to have any alleries which also makes me very happy. Um, but yeah, there's basically nobody who can work on open source software and. Never get anything in return for it and do that for longer than a couple of months so it is interesting I think that for both buoyant and for investor labs having the open source product and working on an open source product that they now do not own.

12:07.61

Dave Sudia

Yeah, for sure.

12:24.39

Flynn

That's another thing that I think gets lost in translation. A lot buoyant does not own linkerd and ah I may have just lost that buoyant does not own linkerd Ambassador Labs does not own emissary anymore and so both of those companies are spending a fair amount of Bandwidth working on products that.

12:27.75

Dave Sudia

Um, yeah.

12:35.65

Dave Sudia

Yeah.

12:42.98

Flynn

They don't actually own but that seems to work out pretty well because it gives us on the 1 hand it gives us the ability to provide something to people that helps them and it gives us a way to talk to them about ways that they can support us in giving them something that helps them. So. Seems to work out. Okay, the yeah I know people who are very very touchy about not wanting to even mention the paid stuff while they're talking about the open source stuff and I am not that touchy I am generally perfectly willing to talk about the paid products. Well.

13:02.31

Dave Sudia

Right on.

13:20.41

Flynn

Like you know doing a workshop for the open source stuff things like that. Yeah.

13:22.62

Dave Sudia

That's why I felt safe asking. Um, yeah so I think on that note ah you know? yes so you were the original author on emissary and you're still a maintainer. Ah I'd love. Didn't know what you think of you know I.

13:27.31

Flynn

Yep.

13:40.10

Dave Sudia

Were just you and I were just speaking the other day about the maintain track talk for emissary coming up at Kubecon and I've you know, just where do you? Ah, one of the things we talked about was just it's stable. Um, and and I'm going to I'm asking some questions here that set the groundwork for some questions I'm going to ask later but ah.

13:50.27

Flynn

Um, yeah.

13:56.90

Flynn

Ah, great, Um, all good.

13:58.23

Dave Sudia

Ah, no in a good I mean in a good way. It's like so for everyone else that for the and for listeners reference I have been using emissary gateway since 28 our emissary ingres since 2018 when it was just ambassador api gateway and ah and so and the.

14:10.94

Flynn

That means that that means that Dave was using it back in the really early days when it was a it was a much it was a much rougher much more difficult to use product than it is now so Dave has my sympathies.

14:15.44

Dave Sudia

Oh the really early days. There were 5 people.

14:24.56

Dave Sudia

Yes, but even with that I will say that it's you know as far as Kubernetes tooling goes which will set the bar low. Ah, it's really nice to use. Ah you know? yeah.

14:33.35

Flynn

So that's a really low bar Dave thanks I think.

14:41.89

Dave Sudia

Ah I was you know? no I mean the api is good. You know like the the the interface is good and um and so yeah I don't have you seen. But I think to that the to that thing of like it was rough. It's better. Have you seen it evolve. Um, where are we at now.

15:00.11

Flynn

Um, actually did a whole talk on this at kubecon in Barthalona which I think was that might have been 2018 actually I think it was um.

15:11.51

Dave Sudia

Um, yeah.

15:16.18

Flynn

And we should probably find a link to that and put it up but 1 of the really interesting things about that was realizing that I think I actually ripped off kierkegaard in the talk and talked about how you know you can only understand this stuff backwards but you have to write it forwards and. It was very interesting looking back and kind of realizing like oh yeah, we started off doing this period where we were experimenting trying to figure out what would work for people and then suddenly we realized oh crap people are using this now. Do we need to do you know? and so the.

15:50.16

Dave Sudia

It's okay, you put Alpha in the api title. So you know we knew it was time to put it in production.

15:52.75

Flynn

Well, yeah, so clearly we're safe right? Um, yeah, exactly when't you get to 0 dot 5 baby? Um, the first release of emissary of you know ambassador api getway at the time. Um, the first release was actually. Early in 2017 and there was a long period of frantically making it so that it was something that people trying to use it in production could rely on even even things as basic as.

16:25.65

Dave Sudia

Yeah.

16:30.56

Flynn

Oh Maybe we should run tests in Ci You know things like that. Um, because you know I mean very few software projects. Those of you who not worked on open source software very few of them start with people sitting down going. Oh yeah,, let's do Ci first Because. You kind of first have to have something that you know enough about to be able to test and that's another dirty little secret of software is that when you start off the developers don't understand the product very well either. And so um, after that came a long period of basically.

16:51.60

Dave Sudia

Um, yeah.

16:59.74

Dave Sudia

Um, yeah.

17:07.47

Flynn

And honestly, this is kind of the position. The Gateway Api is in right now of trying to go through and add all those things all the features and functionality that people need to be able to use your product and anger and then after we got to that point we could start thinking about going through and. Trying to polish off the last of the rough edges and things like that. So at this point Emissary I think is in a place where there are features that would be nice to Add. Um, but I think it has everything that people need to be able to use it as a kubernetes.

17:26.12

Dave Sudia

Sure.

17:43.57

Flynn

Api Gateway I think it has all of those things and I think that it works fairly well for those things. Um I should probably also caveat that last bit with one of the really interesting things that happens when you are especially an early developer.

17:44.73

Dave Sudia

Um, yeah.

18:01.00

Flynn

My mental image of Emissary is always that it's this piece of software that doesn't work very well because the only things that I ever work on in the code are things that aren't working and so I have to stop and remind myself actually.

18:15.81

Dave Sudia

That's funny ton of people run millions of requests per second through this thing. Yeah.

18:19.50

Flynn

If I go back and look at the other areas of the yeah you know we had a guy we actually had a guy once walk up to us at kubecon and say oh you guys do emissary and we went. Yeah, then you do use it. He says yeah I run half a million requests per second and we went wait. What? um you.

18:33.45

Dave Sudia

It works that well and then.

18:36.68

Flynn

Talk to us about how you did that. How does that work you know? yeah because it was you know you you tend to hear as a developer you tend to hear from people who are unhappy about your product that because the people who are unhappy are the ones who file bugs and come to you with issues and open up discussions going. Oh you should really do this. It's.

18:46.67

Dave Sudia

Um, yeah.

18:51.43

Dave Sudia

Right.

18:56.43

Flynn

1 of the things I love about kubecon is that you get to talk to people for whom it's working well and it turns out there are a lot of those people that was really cool.

18:58.58

Dave Sudia

Yeah, yeah, that is that yeah whenever you create something and you get to see someone else. Enjoy it or use it or be happy with it That is always one of the best feelings for sure. Yeah.

19:14.45

Flynn

It's a pretty awesome thing. Yeah yeah, um, and that is you know that that idea of emissary kind of shifting from. Oh we've got to add more stuff to it into the mode of oh there are some rough edges that we should look at right.

19:17.66

Dave Sudia

Um, so.

19:30.94

Dave Sudia

Um, yeah, yeah.

19:31.49

Flynn

Ah, that's a pretty significant transition actually and it's It's interesting to kind of look back and realize oh yeah, that happened a little while ago that was another thing about software realizing You don't see these transitions until they're in the review mirror.

19:45.58

Dave Sudia

Yeah I know what I like best about back. This is just a thing that I just realized as we talked here about like everyone complains about yaml. Okay sure yeah sure and and all the Aml that we have to write.

19:56.26

Flynn

Yeah, that's good yaml sucks.

20:02.23

Dave Sudia

But I was just thinking as we talk about this being a project that is stable and has a stable api and doesn't really need much more I Just realized no one can go through and update the Ui with new colors and a new logo and everything you know like there's no one. Yeah like because I use this budgeting software and I like lost.

20:17.80

Flynn

That's true.

20:22.23

Dave Sudia

The icon on my phone a while ago because they like.

20:30.00

Flynn

Ah, because they could.

20:35.50

Flynn

Okay, that's cool. Yeah, that's cool.

20:56.74

Flynn

yeah yeah I mean I I am biased I'm obviously extremely biased about the getbasor I O C Rds because I invented a lot of them. Um, and there are absolutely things about them that I don't like all that much. But ah. But I have always been pleased about the fact that you can do basic mappings in a small number of lines that just kind of make sense and you don't have to think too hard about them. Yeah.

21:43.40

Flynn

Which is really cool.

21:57.17

Flynn

Not a lot about it.

22:07.62

Flynn

Great.

22:22.30

Flynn

Awesome! I'm glad to hear that yeah one other and sorry one other interesting thing about that kind of getting back to the teaching aspect of things and the tech evangelism thing and you know people. People outside Ambassador labs like you know people we would go through and talk to at Kubecon and such would fairly regularly say things like you know? Oh yeah, the code on this seems like it must have been really hard and many times they were surprised to hear that now actually the code in emissary is mostly fairly straightforward. There are There are some big notable exceptions to that statement but mostly fairly straightforward. No the hard part of emissary was always coming up with a set of semantics that matched up with envoy semantics but that we could explain to people like you know your new grads at Uptree@upcheeve. So that the people trying to use. It could understand how to use it and be able to use it. That was always the hard part and you know I don't think that software in general is it's unclear to me that software in general is going to be making that part easier anytime soon. It is clear to me that a lot of projects don't think very hard about that part.

23:44.68

Flynn

We might like.

24:05.35

Flynn

A fine choice.

24:36.50

Flynn

Yeah.

24:45.49

Flynn

Again. Yeah.

25:29.53

Flynn

Yeah.

25:42.80

Flynn

Great.

25:52.96

Flynn

Yeah.

26:17.99

Flynn

But they were there right.

26:39.93

Flynn

We still? Yeah, we um, every so often I'll get questions about oh I need a good kubernetes one I want and to this day I don't have anything really good to point them at which is actually really frustrating and we should do something about that. But it is interesting to take emissary and linkerd and look at them through that lens because for both of those tools that was an intentional choice in design now in emissary's case it was almost a default. Intentional choice because you know it's not like all of us were Kubernetes experts when we started off and so we started off building something that we could use to solve real problems for us in Kubernetes and very early on. We came up with went went through this this is in like June of 2017 I had to go and talk about emissary or talk about ambassador api gateway at at an istio day actually an istio day conference that was supposedly talking about ingress and I had to figure out how to talk about what was actually different. Between istio's ingress and ambassador api gateway in a way that wasn't just well. We're better and that was where Jane the the app developer persona came from because I realized that was that was.

28:12.50

Flynn

Set of choices that we were doing was imagining this very busy developer who viewed all this kubernetes stuff as friction and we had already at that point and kind of embraced this idea of okay, well if we look at this from Jane's point of view. How well does it work. Um, Linkerd had a similar sort of thing except that with linkerd the linkerd that everybody knows today is linkerd 2 not linkerd version 1 which was a very very different product. And linkerd too especially with the experience from linkerd 1 the designers sat down and went no we are going to make operational simplicity a paramount concern where the point of a service mesh is to reduce complexity in your world. It should not be. Adding more complexity in your world and so they kind of approached it from that direction rather than doing the persona-based thing but you get to the same place which is that you need something that you can gain benefit from if you're a 4 our-person startup with no spare cycles. And you don't have the budget to hire an army of sres. Yeah, this is yeah.

29:47.23

Flynn

Yeah, yeah.

29:53.29

Flynn

Move.

30:09.34

Flynn

Yeah.

30:42.79

Flynn

Yeah.

30:49.37

Flynn

Bricks.

31:12.44

Flynn

Yeah, the the idea that your apis form an interface that humans have to use and therefore they are a user interface is a thing that does not seem to get a lot of attention yet. And that can be very frustrating.

31:41.22

Flynn

So you you don't get good results. Um, if we want to be a little um, a little over the top about it. We can point out that well the way I've often put this in internal conversations is that yet. You know developers are not normal human beings. Um, obviously again, you know said with all the love in the world because I spent decades as a developer. But but yeah, every developer should have the experience Of. Having to support an end user who has no expertise in how your product was built all they know about is what they want to do and they know your product and they see its documentation every developer should need to support somebody trying to get something done with their product because it is an Eye-opening. Experience. It is rarely a pleasant or comfortable experience. But it is definitely eye opening. Yeah.

33:08.88

Flynn

I Think it it.

33:22.91

Flynn

No well that's that that is very fair I think one of the things that makes the cloud native world. Well computing in general but definitely the cloud native world is made very complex by the fact that it moves really really fast and things change a lot. Um, the biggest stuff that we rolled out and we demoed some of this in Kube crash this if anybody is interested in Kube Crash go to kubecrash io and you should find recorded sessions there. Um, but.

34:00.88

Flynn

1 of the things that we did in this one was we demoed having ah cockroachdb and linkerd and emissary all working in the same multi-cluster setup to do a multi-cluster application. Um, because this is also something I think is often missing from the industry is. This whole concept of maybe we should talk about how to do things when more than 1 product is involved as opposed to just doing the toy setups with one product. There will be an ebook for that Kube crash just like there will be an ebook for the previous Kub crash and 1 of the things I'm kind of looking forward to in the. Multi-cluster ebook is getting to wrap in some of the other products that we didn't really have time for so that we can do more of the the real world set up there. But in that one we had. 3 clusters which were working as we didn't actually set them up in these regions. But the idea was that we had 1 in you know Eu West or sorry us west like Sanford let's go 1 in us east one in Eu Central and all of them could share a database all of them could do things emissary. Arranged it so that if you were a european user and you'd logged in accidentally hitting one of the us emissaries. You'd get redirected over to the european one so that we could be talking about gdpr and all that kind of stuff and linkerd was you know one of the really fascinating things when you look at the.

35:35.40

Flynn

At Linkerd and an emissary in those is that um, actually this is true, pretty much every time you have emissary and linkerd working together the way emissary and linkerd work together is that you put them both in a cluster and they just work. Um, so. Yeah, you don't you don't have to do anything magic to tell emissary. Oh. You've got a linkerd you just put it in the mesh tell it and tell it to route and you're done um, same thing. You don't have to sell linkerd anything special about oh this is an emissary you just it's just another meshed workload. It's great. Um. So all of the communications where emissary needed to cross clusters emissary just routed to a workload and let linkerd worry about the cross-cluster business cockroach likewise just would talk to nodes and didn't realize that they were in different clusters necessarily There's database setup. You can do under the hood to tell it things like hey you're a node in this region. So do do compliance this way you know, but that showed off ah a couple of features that I I now think of as second nature but which are actually. Pretty advanced. 1 of them was so yeah, sorry being very articulate here. You can tell linkerd prior to version 2 point 14 did multi-cluster by.

37:04.60

Flynn

Basically having gateways in each cluster and then it could mirror services and it would route things through the gateways to the other cluster and this works very nicely. Um, it does introduce a little bit of latency. It makes it considerably trickier to know the ah the mesh identity of the caller if you have to cross clusters. And for both of those reasons we introduced what I what I tend to call flat network multi-cluster where if you have connectivity such that a pod in 1 cluster can talk directly to a pod in another cluster then linkerd gets to just get rid of the gateways do service mirroring for you and talk directly across and. That's actually really cool because we can do things like mesh level authorization across clusters without having to spend a lot of brain power on the fact that it's you know that that identity is in another cluster. Um, but the other thing that was demonstrated in kuberash that. Didn't really make a big deal about is that the old way in the new way actually coexist seamlessly in the same multi-cluster setup. So for various reasons dealing with headless services cockroach was actually using the old gateway mechanism and emissary was using the non-gateway mechanism at least when we did it. You know when we had that. Um, the cvo clusters we were using we actually had emissary using the old way as well and Dave probably never noticed that at all. Yeah, it's yeah and the thing that I really like about this one is.

38:35.21

Flynn

If you sit down and you talk to the engineers at buoyant who did the multi-cluster implementation. Oh my god the rabbit holes are not to be believed. There's just wow I've had a number of discussions on a couple of things as we. Especially as we start going through and thinking about things like mesh expansion where we want linkerd to be able to extend the mesh onto a virtual machine for example or a real machine outside your cluster and at that point. Whether your workload is in Kubernetes or not all the Kubernetes part of your app is just going to talk to it as if it were in the cluster. It'll be really nice. Um, but the amount of networking detail and knowledge that is buried in all of these things I just described is not to be believed. But. The user interface is you type linkerd multi-cluster link and then you put a label onto a service and it's done which is nice in much the same way.

39:49.96

Flynn

We should talk about that afterwards because that has that's long. Been a thorn in the side of various people and I actually really want to talk to you a little bit about Okay, what what should that user experience look like what is okay with you and not right.

40:13.60

Flynn

I hundred percent we will be having those conversations. Um, the other thing I was gonna oh wow I used it this morning.

40:29.49

Flynn

Um, yeah, yeah, um I also installed emissary this morning just so you know, um what was the other thing. Ah oh yeah, we talked a little bit ago about how for the most part the code inside Emissary is fairly straightforward, but.

40:47.33

Flynn

For everybody who's listening and who is wondering why I said for the most part. Ah yeah, if you want to go and look at the code that emissary uses for Kubernetes Service discovery across multiple sources and things like that not straightforward and. The other piece that's really really not straightforward in emissary is one that often surprises people. It's ah the one that takes multiple mappings and figures out whether they're talking about canary routes for the same workloads or whether they're talking about 2 completely distinct things. Because we we intuit that rather than requiring the user to say it directly because users don't like having to say that directly.

41:33.99

Flynn

A Gateway Api is not very straightforward.

42:03.30

Flynn

Has it.

42:13.98

Flynn

Yeah, yeah, yeah.

42:32.17

Flynn

Only a factor of 3

42:56.36

Flynn

I would I would like for Gateway Api to be something that is extremely usable by everybody in lots of different fields. Um, and.

43:09.50

Flynn

As far as biases from my perspective Go I'm one of the codeleads for the Gamma project which talks about the Gamma initiative which talks about gateway. Yeah Gateway Api for service Mesh Um, and.

43:25.55

Flynn

That has resulted in me spending a lot of time with the code leads of gateway api and in fact I was there for the api review for gatewayapizero.eight and for gateway api one zero kind of in that in that gamma e capacity. Um.

43:44.10

Flynn

And I regularly feel bad about the fact that when gateway api got started I was 2 heads down with emissary to spend a lot of bandwidth and now I am often coming up with things and going hey guys this might be a problem and they're going. You couldn't have said that two years ago Flynn come on. You know it's like I'm sorry I really am um, it's it's a hard problem. It. Yeah, the thing so in the spirit of I feel like people coming to the living on the edge podcast. Ideally would be getting real information from people working with this stuff as opposed to a lot of political spin or something like that. Um, and in the spirit of trying to to be straightforward and honest about where I think the gateway api is um.

44:40.40

Flynn

Yeah, there are a ton of things that you just can't do in Gatewayapi yet that I think are necessary for an api Gateway product to be able to do um maybe the most obvious answer or example there is you know retries. Not part of gatewayapiapi yet. Ah.

45:03.87

Flynn

Yeah, one web sockets.

45:15.21

Flynn

Yeah, there's.

45:26.96

Flynn

Like you have to.

45:35.60

Flynn

Right? Oh we had many opinions about that when we talked about it in the Envoy Gateway meetings. Yeah, um, yeah.

45:48.39

Flynn

Yeah, you have to do something because it's tables. Yeah, yeah, exactly you can't with all the best intentions in the World. You cannot actually deploy an api gateway in production if you cannot do things like Web socket upgrades if you want them. Or you know? yeah you know all that stuff we have Timeouts now. Um you know, but yeah, and so you know envoy Gateway is having to think about things like well maybe we'll just do retries as an http filter. Um, and.

46:27.80

Flynn

Linkerd is in exactly the same situation because we need to be able to do retries and we're probably going to need to do it as an http filter which kind of makes me sad. Um, one of the official takes from the gateway api group. Is policy attachment. Um I finally may have figured out a way to articulate some of my feeling about policy attachment which is that ah so in emissary land we had. Jane the application developer and we have Julie in the opsky gateway api borrowed that role-based concept from emissary and extended it to have Charlie the cluster provider Ian the infrastructure engineer and Anna the app developer and. The really interesting thing about policy attachment is that it's probably impossible I used to say definitely impossible but I'm going to back off a little bit now and say only probably impossible to teach on a how to use policy attachment because. Yeah, yeah, thank you.

48:03.94

Flynn

Britain.

48:38.86

Flynn

No no, they don't yeah.

48:55.13

Flynn

I was helping out. Ah I was helping a customer with a linkerd. Update update that upgrade over the past couple of days where the people that we were on the phone with had no permission to talk to the cluster what they had permission to do was configure ancival jobs. Yeah. Yeah.

49:15.70

Flynn

1 would hope. Yeah yeah.

49:40.46

Flynn

No.

49:48.81

Flynn

And yeah, and the thing so I'm going to talk about a little bit more about why I think this is problematic before we talk about why in the world people would have done it this way because they have really good reasons for it. Yeah, but um, yeah, you get into some really nasty issues there particularly because if we remember for Ana and for Jane all this Kubernetes stuff is friction. They are focused on their business problems. They don't want to go off and learn how Kubernetes works. Even if they do want to they have too much work to do it so they're never going to be Kubernetes experts. But if you're not a Kubernetes kubernetes expert tracking down all the attached policy is going to be very very difficult maybe impossible if you are somebody like Charlie or Ian or. You know Julian for the ambassador labs folks. Those people can be Kubernetes experts and they certainly can go through and learn how all this stuff works. But the flipside of that one is that you know Charlie and Ian are going to be working with things like gateways and gateway classes and stuff. So. Gateway api controls those resources and we should be able to just modify them to contain the policy stuff that we want. Um and so you know yeah, saying things like yeah I tend to believe that we should not be using policy attachment for table stakes features like.

51:23.20

Flynn

Timeouts or cores or you know web sockets because the people who are going to need to mess with them. You know Anna and Jane are exactly the people who will have a miserable time trying to mess with policy attachment now I said I was going to talk a bit about why people would have done it that way because they have really good reasons for it. They do have really good reasons for it. It's a really really hard problem. Um, the biggest challenge here is that some of these things some of these policies. The best place to attach them might be on something like a service resource.

51:57.80

Flynn

We can't change the service resource quickly and because that's a change to Kubernetes core. We measured the timeline for getting change like that changes like that done in years. Um, and you also have a you also get a challenge with um.

52:18.98

Flynn

Broadly speaking you can have a challenge with as you try to pile more and more stuff in there and you want to have the status of a given object reflect things that are going On. You also kind of get into this fascinating scaling Problem. So. As we talk about this I think that it is important to point out that no, we do not think that the gateway Api people did this without thinking about it. They were dealing with a difficult situation and trying to come up with something that at least had a chance of working but it yeah.

52:55.60

Flynn

Um, yeah, it's very much a reaction to the fact that getting stuff in kubecor is just crazy hard. Um, um, that's an interesting position for Kubernetes to be in as a project. Actually and that's one that kind of worries me in some ways really actually it worries me and it doesn't kind of worry me or it just worries me. Um, yeah, it's it's a scary situation in some ways. But so the end result of all of this is that. A lot of the projects trying to use gateway api are in a position where they are they must work ahead of the standard and figure out ways of going forward and that makes gateway api kind of not straightforward.

54:13.38

Flynn

Yes.

54:35.30

Flynn

I think that's about right I have said that out loud in public spaces before that. Yeah, so isn't it nice to know that you're starting off your podcast episode in a nice low controversy manner. Yeah, yeah, no I mean so so the people we're talking about there are Rob Scott from Google and Nick Young at isoveent they are 2 of the the leads of the gateway api the third third one is shaynut at kong they are all. Great. Great people who are doing really amazing work on a really really difficult sort of thing to work with and you know it's difficult because they have to put up with me. So but yeah, it's in all seriousness. It's a tough problem. Um, there are a lot of ways in which so you know looking at this from linkerd's perspective I would like to see the gamma initiative succeed and for there to be a standard so that we can talk about you know like we already. We already get advantage from things like Argo Cd for example, Argo Rollouts can do rollouts based using gatewayapihttp routes to do traffic shifting between your stable version and your canary version. We demoed earlier today that you can use an argo rollout and you can use linkerd 2 dot 14 and you.

56:11.20

Flynn

Don't have to go and install a bunch of extensions and jump through a bunch of hoops. You just tell Argo used the Gateway Api you tell linkerd used the Gateway Api and it just works. It's a thing in Beauty This is this is what we want to see. Um.

56:27.18

Flynn

It Yeah, it should yeah exactly.

56:45.84

Flynn

I think gateway api api yeah around. Okay, it's Apis all the way down.

57:17.40

Flynn

Yeah, yeah.

57:24.19

Flynn

It may very well. There's a there's an ongoing discussion that shows up every so often of who is Gateway Api really for and the.

57:44.33

Flynn

But if you look at the history of it. The initial work focused much more heavily on Charlie and Ian than on Anna and you can so definitely ask a question of okay. Should Ana be using it directly or should Anna be using something else now I'm biased when it comes to that question in that the whole of my career in cloud native has been focusing on Anna and Jane and not on. Folks down in the infrastructure world because ultimately nobody runs this stuff just to say they're running it the humans using these products are using them because there's something that they need to get done. And if we are not doing things in service of them getting their stuff done I think we're doing it wrong and ah, that's not to say that I think that the charlies and ians of the world are not important. They are very important but we should be focusing on the Annas and janes because. Ultimately, they're the reason that all of these computers are running in the first place. So.

59:14.57

Flynn

Ah, it works for me. Ah.

59:19.39

Flynn

No, no, definitely not.

59:30.85

Flynn

Ah, they should talk to me about linkerd and about emissary and about gateway api and bio gamma and about Anna and Jane really is what that comes down to um where they will find me I will be at. Most of the time I will be either at the linkerd booth in the project project pavilion or the buoyant booth in the sponsor showcase or whatever they call it. Yeah, the sponsors. All um I will be delivering the emissary maintainer track talk with hamakudi of ambassador labs. I will be doing a talk about adopting the gateway api without ruining your reputation. Um.

01:00:18.95

Flynn

Ah, um, sorry dude im ah it was actually my coworker mattte who put it in but mate is not going to be able to make it to kubcon so I'm going to actually deliver it that is on the. That's on Thursday at 11 a m the emissary maintainer track talk is on Tuesday at four thirty and Tuesday at five thirty I will be doing a panel with some other service mesh folks that's moderated by Keith Maddox of Mike. Microsoft and it'll be me ah Linsun from oh I'm going to get all this stuff wrong lane soon is from solo Thomas Graph is from isurveillant and John Howard is from Google and we will be talking about. The down and dirty being in the trenches of how service meshes are awful and how we hate everything and all that or something like that. Yeah.

01:01:28.48

Flynn

And I will definitely be finding you at Kubecon. Yeah.

01:01:51.81

Flynn

And here's hoping you sell out.

01:01:59.52

Flynn

Ah, now you know I am going to say I've heard in the past that the kuerokie party you've been up till like 3 or 4 in the morning and completely blasted when you show up at kubecon the next day so I will point out that for those of you who want to be 1 of the coolest people and go to Kueroki just recognize. Certain amount of occupational hazard.

01:02:28.45

Flynn

Ah, thanks for having me we'll we'll do it again. Sometime sounds good.