I used the old CUE-based version when it came out, and was really excited about it. I liked it, and enjoyed working with CUE, but the API was clunky and incomplete.
Then they completely abandoned not just the CUE frontend, but CUE altogether (while strenuously denying that they were doing so) for a GraphQL-based rewrite that focused on letting people use popular general-purpose languages to construct their workflows. The initial rollout of this was not feature complete and only supported imperative languages (Python and TypeScript, IIRC), which I didn't like.
Instead of porting everything over to all their new interfaces, I hopped off the train and rewrote all of our portable pipeline scripts in Nix, via Devenv. At the time, I'd never used Devenv before, but getting the work done that time still took maybe a tenth of the time or less. More than anything else, this was due to not having to fuck around with the additional overhead Docker entails (fussing with mount points, passing files from one stage to another, rebuilding images, setting up VMs... all of it). I got the reproducibility without the extra bullshit, and got to work with interfaces that have proven much more stable.
I still think there's a place for something like Dagger, focused just on CI, perhaps even still using Docker as a deployment/distribution strategy. But I no longer trust Dagger to execute on that. I think a proper external DSL (probably special-purposw but still Turing-complete, e.g., Nickel) is the right fit for this domain, and that it should support multiple means of achieving repeatability rather than just Docker (e.g., Nix on bare metal and Podman, to start). An option to work on bare metal via reproducible environment management tools like Nix, Guix, or Spack is a valuable alternative to burdensome approaches like containers.
I haven't looked at Dagger in several months, but the other big piece that is missing for portable CI workflows is a library that abstracts over popular CI platforms so you can easily configure pull/merge request pipelines without worrying about the implementation details like what environment variables each platform exposes to indicate source and target branch.
Idk anything about all the AI horseshit; I was off the Dagger bandwagon before they took that turn. I don't know if it's serious or a nominal play to court investors. But that kind of pivot is another reason not to build core infra on top of the work of startups imo. If the product is 70% of what you want, you have no way of knowing whether filling that 30% gap is something the maintainers will suddenly pivot away from, even if their current direction looks aligned with yours.
I'd recommend considering tools in this space only if (a) they're already close to 100% of what you need and (b) they're open-source. Maybe you can relax (a) if it's really easy to extend the codebase (I find this to be true for Devenv's Nix modules, for example.)
Do you have any examples of your Devenv workflow you can share? I took a look at Dagger and really like the concept, but I'm trying to figure out the limitations/why there's so much negativity in this thread.
I currently manage my development environments via NixOS and Devenv, so if I could just keep that and achieve the same functionality, that sounds good to me.
At the time, Devenv was still pretty barebones compared to now (the rate of feature development has been great in the past few years), so it didn't have much in the way of support for workflows per se.
Instead I used the scripts module to write shell scripts that bring along all of their dependencies. Thus all of our GitLab jobs are just one-line invocations of our wrapper scripts. We currently use a flake.nix-based Devenv and expose the scripts from the Devenv module as flake outputs, then just run them with `nix run`.
Nowadays, though, Devenv actually directly supports some relevant functionality in the form of tasks¹, which can be written in any language, and support dependency relationships and run concurrently by default.
We use those in a few places as well, though mostly in place of older, cruder `enterShell` integration. There, we use them:
- to seamlessly ensure correct state of Git extensions:
- all Git submodules are checked out
- Git-LFS is configured and all large files are correctly checked out
- to ensure valid OpenTofu caches since we use Nix instead of OpenTofu for managing providers
- if .terraform dir doesn't exist, run `tofu init`
- if Terraform providers lock file is older than devenv lock, run `tofu init -upgrade`
- manage Poetry stubs and virtualenvs (most of this is configured behind the scenes upstream, but we do some extra stuff)
- manage Bundler dependencies and binstubs for Ruby projects so executables under development can be involed naively from PATH (much of this is bespoke but upstreamable)
So a lot of those Devenv tasks are things you might do in build or pre-build stages of a CI pipeline. You can define the requisite dependency relationships and then just run `devenv test` or `devenv tasks ...` as your CI job.
What we don't have is any abstraction over the various CI platforms we use (currently GitLab, GitHub, and AzDO, with Jenkins support in the backlog). Unfortunately, for the sake of a better DX for our (internal) customers, we still integrate fairly directly with each CI platform, so we do have piles of vendor-specific YAML.
Our tasks and scripts are currently just Bash, but they're relatively nice Bash because their dependencies are managed with resholve and they're linted by ShellCheck, thanks to some experimental changes I made to Devenv's scripts module.
For our security/quality scans, we do, however, have some logic nestled in those Bash scripts which checks for a CI environment via env vars, then performs differential rather than full scans if we're in merge request contexts. Where the upstream tools don't directly support this, we basically perform "before" and "after" scans and take a diff. This is the direction I'd like to go in the future— all in on this, surfacing jobs in a "native" way be damned.
Unfortunately I can't directly share these examples as they belong to my employer and live in proprietary projects. But what I can do is open a couple of WIP PRs for the upstreamable work (new or modified Devenv modules) so you can take a look at that at least.
Also fwiw I think Dagger is still worth checking out. I just feel a bit burned by adopting it too early in its lifecycle, and generally advice caution about adopting tools developed by VC-backed startups.
