Oh man, I tried to use WARP for a project and it was the most confused, hard to understand product I've seen for a while. After a couple of hours of reading unhelpful support pages and trying increasingly random settings I simply gave up and used tailscale instead, which worked instantly.

I'm sure it wasn't the part of the product you worked on but the onboarding experience, wow, just terrible.

Through OpenZiti into the mix too - https://openziti.io/. Its open source and was designed from the ground up with zero trust, SDN, and deny-by-default principles. It also includes SDKs to allow developers to embed ZTN as part of the SDLC. We also built zrok (https://zrok.io/) on top of it, as a demonstration of a 'ziti-native' app, and being a better Ngrok.

OpenZiti looks so enticing but it is certainly a completely different world.

Are there many big installations of it? With the vast majority of stuff being written for regular networks I do wonder if the amount of proxies/sidecars becomes unreasonable. Probably not though

The biggest installation is a cyber security unicorn who whitelabels the commercial version of OpenZiti (NetFoundry) which they are selling to many large enterprises. They have hundreds of thousands of endpoints deployed. There is also a massive OT ICS OEM is embedding NetFoundry into its industrial routers, PLCs, etc for all types of connectivity, including machine to machine in factories. The same technology is being used for tactical military networks, with drones connecting to servers. Its being deployed in critical grid infrastructure. A hyperscaler is adopting it to replace hundreds of thousands of VPNs as well as build a multi-cloud zero trust offering for server to server workloads.

So we have not finished our job to make OpenZiti the equivalent to Linux in that every just uses it by default, but we will get there ;)

You raise an interesting point. We provide flexibility for ingress/egress... do not like proxies/sidecars, go in either opposite direction, (1) ZTAA, Zero Trust App access with SDK app embedded via SDLC, (2) ZTNA, Zero Trust Network Access via appliance deployments in DMZ/VP/VNET etc (the one you mention is ZTHA, Host Access). Each of ZTNA/HA/AA have different pros and cons based on your requirements and use case.

> A hyperscaler is adopting it to replace hundreds of thousands of VPNs as well as build a multi-cloud zero trust offering for server to server workloads.

This one is the most compelling to me! A while back I was building a small cloud provider and this was the use case I was looking really closely at OpenZiti for.

Thanks for sharing!

>You raise an interesting point. We provide flexibility for ingress/egress... do not like proxies/sidecars, go in either opposite direction, (1) ZTAA, Zero Trust App access with SDK app embedded via SDLC, (2) ZTNA, Zero Trust Network Access via appliance deployments in DMZ/VP/VNET etc (the one you mention is ZTHA, Host Access). Each of ZTNA/HA/AA have different pros and cons based on your requirements and use case.

Ah so this would work if:

1) all the apps were my own

2) I was sitting on top of a major cloud/managed colo (I'm just sitting on top of infra providers like hetzner)

The project I was working on is not so active right now but it's still chugging along (I use it)... I was looking at things like OpenZiti as a k8s CNI provider, or used along with something like Cilium/Calico (but then they'd both promise network security in-between workloads and overlap)

You could use OpenZiti together with Cilium/Calico, there are some distros, eg., https://kubezt.com/ which do that (though in truth, KubeZT has moved to Istio for E-W, uses OpenZiti for N-S. OpenZiti does a lot of things that service mesh technologies do not, for example, extending outside of the cluster (incl. to non-K8S workloads), allowing closing of inbound FW ports, providing a private DNS outside of cluster, removing the need for VPNs, L4 loadbalancers, MPLS, SDWAN, public DNS etc.

Yes, OpenZiti very much focus on private apps, whether COTS or inhouse developed.

Oh, I should note too, while we have a bunch of ways to deploy OpenZiti on K8S today, we are in the process of building/releasing an admission controller and an ingress controller for OpenZiti.

Whats the project you work on?

> You could use OpenZiti together with Cilium/Calico, there are some distros, eg., https://kubezt.com/ which do that (though in truth, KubeZT has moved to Istio for E-W, uses OpenZiti for N-S. OpenZiti does a lot of things that service mesh technologies do not, for example, extending outside of the cluster (incl. to non-K8S workloads), allowing closing of inbound FW ports, providing a private DNS outside of cluster, removing the need for VPNs, L4 loadbalancers, MPLS, SDWAN, public DNS etc.

Yeah Istio is a hard no for me... And yeah I definitely appreciate that OpenZiti does a lot more than service meshes do! I personally try to avoid service meshes (if I were to use one, I'd go with linkerd).

I'm just not convinced that many people need a service mesh -- I haven't really needed one yet, but maybe I'm just not at the right scale/etc.

> Oh, I should note too, while we have a bunch of ways to deploy OpenZiti on K8S today, we are in the process of building/releasing an admission controller and an ingress controller for OpenZiti.

This is awesome -- I really like my current admission controller though (Traefik), it's FANTASTIC. I think moving ingress controllers might be a large lift for people (it would be for me).

> Whats the project you work on?

I don't really work on it actively these days (haven't in a while) but https://nimbusws.com

Looking forward to picking it back up more actively in the future though, for now I use it for some small background services.

Ahh, thats cool. Note, the app embedded capabilities of OpenZiti becomes very interesting in this scenario, for example, our Python SDK working with Boto3 for AWS S3 - https://blog.openziti.io/extend-access-to-a-private-s3-bucke....

I get that. Would need to check if we can integrate Ziti into Traefik in the same way we did for Nginx (https://blog.openziti.io/nginx-zerotrust-api-security) and Caddy (https://blog.openziti.io/put-some-ziti-in-your-caddy), which we did respectively using the C and Golang SDKs. Therefore you wouldn't need to change your ingress, you would just load Ziti as a module in it.

> I get that. Would need to check if we can integrate Ziti into Traefik in the same way we did for Nginx (https://blog.openziti.io/nginx-zerotrust-api-security) and Caddy (https://blog.openziti.io/put-some-ziti-in-your-caddy), which we did respectively using the C and Golang SDKs. Therefore you wouldn't need to change your ingress, you would just load Ziti as a module in it.

Yeah, this is why I was thinking that the easiest way to integrate would be a sidecar (and in general for random web serving payloads). I'm not ruling it out, but this was the major stopper -- it seemed easier to just rely on Cilium/Calico underneath to keep east-west comms encrypted between apps and k8s nodes.

> This is awesome -- I really like my current admission controller though (Traefik), it's FANTASTIC. I think moving ingress controllers might be a large lift for people (it would be for me).

Clarification: You could use Traefik's ingress controller in tandem with a hypothetical OpenZiti ingress controller. You'd set `ingressClass: openziti` on those Ingress resources you wish OpenZiti to handle. Nothing would prevent you from creating two Ingress resources for the same ClusterIP service: one each for Traefik and OpenZiti.

Oh so that allows it to run in-process?

That's cool, I did that for an HTTP forwarding thing a while back.

Yes, indeed, this blog gives a great view on it - https://blog.openziti.io/go-is-amazing-for-zero-trust - using Golang and HTTP examples. My favourite part:

"Now, your server has no listening ports on the underlay network. It's literally unattackable via conventional IP-based tooling. Seriously, stop and consider that for just a moment. By adopting an OpenZiti SDK into the server, all conventional network threats are immediately useless."

Well that's also true for Firezone :)

It's a tradeoff between in-process and out-of-process though. It's nice that Firezone Gateways don't have access to the service's memory space and can't crash the process, but it's also nice that an in-process Gateway equivalent doesn't need to loop through the network to reach its service.

Maybe we are referring to different things when we say 'process'... I am not aware (happy to be educated) of Firezone having SDKs to embed the zero trust overlay running directly in an application, i.e., in the app process and memory.

Do they support this?

I hear you on having 'out of process', that's why OpenZiti also has tunnellers for deploying on host as well as virtual appliances to run in the DMZ/VNET/VPC etc. I was only aware of Firezone supporting those 2 deployment models.

As a WARP user that has experienced a lot of problems, I'm glad that Cloudflare is allowing WARP users to switch from wireguard to MASQUE. I'm hopeful this will solve a lot of our issues.

Hm, is Wireguard getting blocked by middleboxes or something?

Sometimes yes. MASQUE just looks like normal TLS port 443 traffic so it's less likely to be prevented. Other issues we had were just bugs with the WARP wireguard implementation regarding how the tunnel is built in Windows. I imagine it works better on Linux or MacOS which is likely what the developers were running when they designed WARP.

Depends. MASQUE implies QUIC and a lot of coporate networks still block QUIC, forcing a fallback to TCP (for web browsers).

Not sure how they want to address this in case of WARP?

Nevertheless, MASQUE is some pretty exciting development in the network space.

Ah that's tough because Wireguard being UDP is a selling point for us at Firezone

Ah yes, don't get me started on all the fun edge cases involved in supporting cross platform network stacks and all their subtle tunnel API differences :-)

We're trying to stay focused on keeping policies easy to define and manage so that access management is... manageable as you scale. That and the fact data doesn't go through the cloud / intermediary have been a couple of the reasons customers say they prefer us.

Definitely an exciting space to be building in!