As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management.
Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason.
Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can't be stopped from having the newest version of things, and I think that's a bad idea.
15 years ago, when a project would kick-off, as a sysadmin I'd be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure.
Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I'm not calling pip out as particularly bad; I'm calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Ruby is just as bad; the famous (and I've experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it.
Devs are chasing a moving target, and devs who haven't been sysadmins may have trouble seeing why that's a bad idea.
As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that.
Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.
If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can.
> They aren't kids playing with toys that we need to nanny over
In my experience (university), yes they are, and they should do that at home.
Why do you need the latest bleeding versions in the first place?
In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance.
With my developers hat on, bumping to the next version mid-process reliably introduces more friction than is worth it. People think the next version solves that one weird issue but ignore that it introduces two new ones and that the software must be changed to fix five new incompatibilities.
But the solution reliably is to just not use that weird feature that caused the bug in the first place, and think what a clean solution would have been. And guess what, the result is a cleaner and more compatible code base. It's a tip that works for me again and again: If there is friction, think - before spending the next hours with an update that will soon lead to new problems.
It's great that you can for example compile Linux without too much friction. It's great that arcane shell scripts can run on any system. Stability (in a compatibility sense) is not a nice-to-have, it's basic sanity.
Stability, sanity, all that is amazing, and a must have.
But also bug-fixes, security improvements, and performance improvements are wonderful too, which tends to come with using up-to-date dependencies.
The problem with the latter, as you mentioned, is when it introduces breaking API changes and is wholly not backwards compatible. This is not a "kids playing with toys wanting to experiment problem" this is a bad software problem, which is why I like Go, and why I liked Java when I was doing it full time. If the language you use has backwards compatibility as a first-class citizen, most likely the package authors will act that way too, and then the maintainers, and eventually the developers. Limit your software choices to those who care about not breaking everyone's shit every 2 weeks. Heck even when I write my own API's now that I know only my company is going to use internally I am thinking about this.
Backwards compatibility is seriously under appreciated. When I tell developers to ensure that their changes are backwards compatible, they tend to look at me like I'm green.
I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.
Unless you have a situation where it's impossible to maintain, not insisting on it is pure self interest.
I do not understand the disconnect that developers have with understanding all of the benefits that it brings.
Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday.
If you were speaking to seasoned C and DB developers about stability in the tools and the platform, you'd be preaching to the choir.
This gets to the complaint that so much of the open source ecosystem gets to version 0.8.6 (whether it's named that or not) and then completely rewritten "this time the right way". That's not actually a good thing.
> It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8?
> Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.
Backward compatibility has real costs. You cannot restructure your code base as easy, you cannot deprecate bad ideas, cannot extend it as easy more and so on and so on. Sure, it also has real benefits (as you've stated), but missing the disadvantages while only highlighting the advantages is not a useful approaching, it only shows "your personal preference".
> Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.
No, the former results in a bloated code base full of old legacy crap that no one understands and is afraid to touch because it might break. You have to insert weird workarounds because that bug is now a feature to some idiot and you provide backwards compatibility so it lives forever now.
Has software quality increased now that everyone is refactoring and rewriting everything for every release?
I don't think so.
Its just plain arrogance to believe your are a better developer than the guy who came before. Having some fear of breaking the code base is healthy, the same way that having a little fear that the chainsaw is going to cut off your leg makes you safer.
I like your answer. It's civilized, balanced, and I agree with every word of it :-)
I'll add as an anecdote that I do follow your practices (limiting dependencies). It works wonderfully on Debian stable (most of the software there is now >2 years old, the next version has just been soft frozen). I have the occasional package pulled from testing: I recently toyed with Perl6. And currently I use a newer version of python3-sphinx for a nicer doc syntax but I could do without. It causes no headaches at all.
For one thing, if there is any place to treat software like a toy and to play around with the latest version, it would be a college or university.
I don't particularly care about having my software on the latest version. I personally prefer using the old version for six months while the newest version gets the bugs worked out of it.
I know sysadmins value reliability and security, but it's really frustrating when every upgrade takes dozens of hours of work to approve. Questions like "What features do you need in the new version" miss the point. It isn't about the features of the software, it is about maintaining a modern code base.
Upgrades always have the potential to break things, but when you keep up with the upgrades it is easier to achieve the stability and security goals the sysadmin wants. When you upgrade often, it is easier to read the documentation and find where changes might break something, and when things do break it is easier to fix them. Upgrades that jump over several versions at a time are a nightmare to debug, and it creates a lot of technological debt that you have to work out later.
Ultimately, sticking with a version of software because it works is trading a little stability now for an absolute mess down the line.
I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.
I don't think this thread is about "maintaining a modern code base" at all. Whatever that should mean -- My impression is you've fallen victim to the hype train.
In my perception the thread is about always catching up with the latest and greatest. Would you say in all earnest that my code is not modern because I make a point of developping against solid standards and not constantly longing for things that are not in my distribution (the software there is usually 0.5-2.5 years old)?
You can check some of my code at https://github.com/jstimpfle. Is it "not modern"? I'm a reasonable but not outstanding developer, and not saying that everything will work on your computer (since I'm usually the only tester) -- but I'm pretty sure I can get everything there to run on your computer with minor effort.
> When you upgrade often, it is easier to read the documentation and find where changes might break something, and when things do break it is easier to fix them.
No. Breakages are less frequent because the software is not brand new, and they are better known because all people using the stable release are on the same version. Documentation comes with the distribution, but I don't have any problems googling it by giving the version string either.
> Upgrades always have the potential to break things, but when you keep up with the upgrades it is easier to achieve the stability and security goals the sysadmin wants.
This thread was never about security and I don't approve. I don't think you are familiar with the concept of a stable release.
> Upgrades that jump over several versions at a time are a nightmare to debug, and it creates a lot of technological debt that you have to work out later.
No. If you develop against solid standards you have less breakage. It's not about incompatibility with the most recent versions. That would be a stupid idea. It's about compatibility with releases other than the latest and greatest. This means not depending on the hot new features that are only in these versions, simple as that.
> I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.
That's fine for an OS, but what do you think business customers would say if you said "sorry, that feature won't be added until the next release in 2 years time"?. That's were tools like pip come in, they let the software move faster, which it often needs too.
> Well for me, all of the libraries I use because none of them exist as packages for any OS.
In conclusion you don't use any libraries that are packaged for any OS.
What libraries?
Also assuming that some libraries you use don't exist for your OS, that doesn't mean that you absolutely need the latest and greatest in a business critical way. So, not approved.
All in all, not too fond of the reasoning and the evidence you provide.
Your logic would prevent any app from just about any non-c ecosystem running. Java, ruby, python, dotnet, rust, go, they all have their own library management and very few of those libraries will be available in an apt repository (let alone a compatible one).
Your policy may work in a university, but you'd be fired from any real business.
You're also making bold claims ("any app from just about any...") that my personal experience can just not validate. It's very easy to write applications without fancy dependencies. Recently I did algorithms, systems and applications in C, C++, python, sdl, alsa, X11, Unix shell scripting, lp_solve, and some web programming in python, javascript, sqlite3. All rock solid and stable - all will probably run on any Linux box from the last 5-10 years (python3 came only in about 2008; forget about the lp_solve bindings and just use the command-line tool).
There are 2050 python3-* packages on my system. Not that I think it's a good idea to use most of them. What's "compatible"?
So what are the libraries you absolutely need? What is this week's secret sauce?
.net MVC, nUnit 3, jquery, knockoutjs, and nHibernate to name a few. I could name a dozen similar tools on the java stack. Pretty soon I'll be experimenting with rust with libui.
>There are 2050 python3-* packages on my system. Not that I think it's a good idea to use most of them. What's "compatible"?
2000 of them are random versions someone made a package for that are unknown to the core team and probably not receiving updates.
Your list of projects sounds like typical academic ones, not tools used by businesses that employ most software developers.
You're also missing the other benefit of these tools, that we develop against the deployed version. There are no compatibility issues because we develop on ubuntu and host production on redhat.
If you consider making websites, and tools and web applications for internal processes, academic... I also did some contract work where I created a server and a client for displaying advertisement media to commodity screens, in a tiled fashion. The tools were all there, C, python, bash, X11, some media libraries, the versions there were all fine (I worked around a bug in mplayer though).
jquery, knockout. I don't know a first thing about bundling dependencies for client-side javascript code (not going super fancy there, don't need jquery, knockout - tried writing single page apps by hand but they are hugely complex) but anyway, isn't that independent from server installations? Don't you bundle these libraries in-tree? If so, it doesn't relate to the discussion.
.NET... It's MS, do you run on Mono? How does the question of requiring the latest version apply?
> Your policy may work in a university, but you'd be fired from any real business.
Nah. If we define "real business" as something with a decent turnover, employing over 250 people, and being in business for over 8 to 10 years; a business that isn't actually in the business of writing software (the majority of what makes up global stockmarkets, or "real business" in most peoples' eyes) then you will find that OP's attitude and policy-making philosophy is right on the money. (Source: Was CTO in exactly the above type businesses for many years)
> Why do you need the latest bleeding versions in the first place?
Because the newest version has several features that we would like to take advantage of immediately?
Look at PHP 7.0 which introduced return types, and 7.1 which introduced nullable return types. These are features I really want in my application, so we upgrade.
I think we agree on the prudence of professional engineers.
The definition of 'professional' is up for debate, but I'd encourage people to weigh in on the following (to IEEE, not me):
- an appropriate engineering education background (ABET/EAC)[1]
- at least four years of engineering experience in your field and under the supervision of qualified engineers
- passed two exams (the Fundamentals of Engineering [FE] exam, which is now a computer-based test available essentially year round, and the eight-hour PE exam)
- kept current by as a minimum meeting your state's continuing education requirements.
Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.
If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can
Well - that's all well and good, when put such that "Gatekeepers" are viewed as blockers.
However, we "Gatekeepers" are the ones that get paged and / or yelled at by a CTO when an application keels over. Not the developers. The developers get to sit in their sandbox (otherwise known as "production" in 2016/17) of ever-changing library versions that were only rapidly tested in QA. Then they play a game of Starcraft II, scan HN and go to bed. When something runs out of memory or crashes in the middle of the night, we get paged. So, hell yes we should be involved in the process.
When I started at my current company the traditional silo between dev and systems was there (although we were allowed to deploy our own stuff) - they managed everything we ran our apps on and we just deployed them to servers they had already configured. Over the past ~3 years we've made a lot of changes, the department manager for our IS team is present in our daily standup calls to relay information between our two teams and we now have a couple separate VMWare clusters dedicated to our applications and VM running on them is our responsibility for the most part. We are the first to get called for issues with our applications, and where necessary we work collaboratively with our systems team to resolve them - we don't throw blame around, it does no good.
I should add most of this is only possible because we have real DevOps people on our team (well, really, it's just me right now - we lost our other and need to hire a replacement still) - not developers who know enough to copy a blob of crap to a server to run, but people who have real skills in both aspects. We are trusted to maintain things because we can do it right, and while it took a lot of work (and some unfortunate infighting) to get to this point both of our departments are working great with this arrangement.
There's still kinks that need ironing, we've not done an adequate job at writing documentation so our systems team can help with some failures (primarily on our Linux VM's, our whole systems team is Windows admins) if we aren't available - but it's on the radar as well as getting PagerDuty set up to escalate alerts to them if we don't respond in time (like having our PostgreSQL data volume fill up over the weekend, not a call I wanted to get at 10AM on Sunday).
So yeah, fix your culture issue, get people communicating daily between your teams, share responsibility for issues instead of placing blame.
> However, we "Gatekeepers" are the ones that get paged and / or yelled at by a CTO when an application keels over. Not the developers.
And that's why people are moving away from that model. It's part of the reason DevOps is being embraced as a model. Developers should be on call to support the applications they build. You get benefits all around.
should is the key-word here. As a sysadmin, I'd love to work closer with devs, especially during outages. Unfortunately every time we bring up on-call, the room goes silent. This is very anecdotal, but IME, devops has just become a way for devs to bypass sysadmins.
I wonder how many companies are doing it right vs doing it wrong? Any anecdotes from a proper devops group?
Then place the blame on the gatekeeper. As a sysadmin, I'd be more than happy with you pointing the finger at me as the reason why you can't deliver a feature. Assuming, of course, that you've run the proper tests and gotten QA's approval.
Honestly. Developers should stop thinking that their jobs is to release features all the time at all costs. That's simply not true and that's counter productive for the business.
And in the end, that's the developer who always gets the decision. Does he ships half assed half finished every single time, or does he take time to do some testing and not break production.
The problem with your characterization of things is companies like the one I'm in where us "Developers" have replaced the gatekeepers entirely. We have no dedicated SysOps, and yet our production environment stays up just fine.
> It removes the underlying need for a Sys Admin to worry about the versions.
No, they really don't: they remove the ability of system administrators to administer versions of software across the total system.
This is bad, e.g. when a new OpenSSL vulnerability comes out (it being a day ending in -y) and every piece of software has to be updated.
> We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.
I am a developer, and I disagree. We are, by and large, kids playing rather than adults making carefully considered decisions. We'd rather use v3.0.rc-1-awesome rather than 2.17.12, because the former is the version that adds an API that saves us from writing twenty lines of code, never mind that it also is untested, unstable and very likely insecure.
We need adult supervision. We need oversight. That's why I argue for using stable, LTS-style distributions, and running against the distro packages unless there is a very good business reason not to (and yes, 'we can't implement necessary functionality in a cost-effective timeframe' is a valid business reason). I'm not opposed to using the bleeding edge when it makes business sense; I'm opposed to developers using the bleeding edge because they like it, and keeping the business in the dark.
I agree with OP and would amend his statement to "15 years ago, when a project would kick-off, [sysadmins & architects would] be invited ..."
From an architectural point of view, microservices take the reductionist approach to system design to an absurd limit, and per my professional experience (fwiw & ymmv) are due to the general architectural illiteracy of the rank and file practitioners in this field.
Right, microservices isn't "architecture". It is - whether they know it or not - an admission that "we can't do architecture so we'll chuck it over the fence and let the ops people worry about how it all hangs together".
Yes the 'no-architecture architecture'. It's very Zen. /s
> how it all hangs together
In case you are interested in rescuing :) young but promising talent in the field, next time you find yourself involved in a discussion about microservices "architecture", point out the realization of a single-node application per this approach, where every function is a process, and the call stack requires IPC, and the 'linker' is considered obsolete and outdated technology.
I once worked on an application that - no joke - comprised 5000 VMs each of which was running one "service" in a dedicated JBoss. It was laughably bad.
At what point do you determine who is responsible for securing the environment? The "Gatekeeper" mentality is stemming from this. There is no clear line in any organization and I see the blame-game all the time.
This is exactly the example I give when I try to explain to someone that managing through personal responsibility as opposed to team and organizational responsibility will grind your company productivity to a crawl.
There are actually generally rules for which employees have responsibility over the vaults in banks. The employees aren't equally responsible. Roles and responsibility are defined strictly. Generally, the lowly tellers aren't allowed the same access to the money as the general manager, and they aren't held responsible to the same degree if money in the vault goes missing.
This has actually been a very insightful response to me.
After reading your comment it now occurs to me that Docker and other container systems are actually a huge organizational tool. One issue I have encountered at companies is keeping the IT and development departments on the same high level organizational incentives to keep political barriers from coming up between them (and conflicts arising).
Containers can help keep everyone's incentives aligned because System admins can focus on the actual administration aspects of the systems and infrastructure (that devs do not need to be concerned about, like vnet layouts and whatnot) while devs can focus on the actual development and deployment without having to have everything confirmed and approved by the IT departments.
At almost every shop I've ever seen, "sysadmins" are also the ones whose responsibility it is to at least attempt some sort of security practice and business continuation. Which opaque, "just run it" containers actively fight against. Did the developers actually audit what they've pulled in as dependencies? Did they make sure that they can be rebuilt if whatever package source goes away? Where is everything documented? Containers, as usually implemented, rather than "keep everyone's incentives aligned", instead damages the ability of the adults in the room to keep everything from falling apart.
(I have been both the sysadmin saying "no" and the developer mad at sysadmins saying "no". But going slower and doing our homework has never, ever hurt me or my employers.)
I've never seen sys-admins verify security of dependencies outside of the major dependencies (like the language VM version). Security wise they have been much more concerned with the security of data storage systems (databases, elastic search, etc...), the operating system, and the network in general.
Security of the application is very much so responsibility of developers, not system admins, as the developers have the best point of view to understand the implications of the software they are developing/integrating with.
If there are routine violations of security at the application level that aren't being caught by developers working with those systems then the company as a whole needs to sit down and make sure the development teams have the proper security procedures in place, because putting a department in charge of security that has all accountability but no power to remedy the situation is a recipe for political fights between departments and a disaster. Proper code reviews and team leads with experience should be able to catch more security issues than sysadmins will.
If your sysadmins are in charge of security review of the application then they have to be in charge of security review of every low level dependency at the individual package level. Otherwise your developers won't think about it because it's not their problem (IT will review and let me know if anythings bad) and it encourages them to lack accountability of the security of their own software.
Isn't it just as bad to keep versions locked for months and years so we don't upset the delicate balance of versions on the production env?
I've worked at a place where they were running PHP 4.4.9 until about 8 months ago. And they were upgrading to 5.4! I get that it was work to convert a lot of the older code base to 5.4, but it was already passed EOL when they were switching to it. And 5.5 wasn't much behind it.
So now in the near future, they'll need to upgrade again (though they probably wont), and they'll probably jump to 5.6, which EOLs in two years (probably two years after it EOLs).
That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use?
I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading?
I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.
why should a sysadmin who doesn't even know Python tell me what version of a library I will use
Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences.
What does a sysadmin have to approve or deny an upgrade?
Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...
Developers tend to fall into the trap of believing we are better than sysadmins but there is immense value in having talented admins around. I have recently experienced this first hand watching someone breeze through server archaeology and vitlrtualization tasks that I struggled through and may never have been able to accomplish in a reasonable amount of time. When admins and programmers recognize each other's strengths and play to them it is a rewarding experience. We just have to realize that we're on the same team.
Also, docker (and containerization in general) is a wonderful thing for both of us. It decouples the fickle apps from systems (also moving targets) and the other apps which are constantly seeking out new and creative version incompatibilities. It makes migration and maintenance a much less frustrating endeavor with fewer surprises along the way.
>Because he maintains that installation and you don't?
So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK?
>Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer).
That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.
>Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily.
Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out.
1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access.
2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service.
3) Automate builds of containers from repo and add automated testing post containerization.
4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container.
In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless).
There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.
I see your point, and like I said I agree this is why Docker was invented and it's the best-in-breed at what it does (namely, being a tourniquet for a self-inflicted wound). My biggest concern really isn't "my problem" since I'm in ops: it's the leftpad worry. I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems".
And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. I guess I still think there's something to be said for the cathedral pace, though.
>I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems".
Well there are two ways to approach this IMO.
One be proactive. Create them a vetted centos or whatever OS environment for them to base off.
The problem is, if you don't keep on top of it as a sysadmin, the developers will just figure out another way to wall you off.
Alternatively, accept that it doesn't matter what underlying OS they use, because a patched OS >> than unpatched and that when done correctly there is minimal exposure even when the service has an exploitable lib due to the jailed nature of containers.
Assuming "latest introduces new problems" too often builds an aversion to patching which can lead to worse issues down the road.
>And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it.
Exactly! The only one responsible for libs are the parties directly leveraging them. Not that developers shouldn't make that info known. It has to be documented to remove the bus-factor of 1, and if it isn't the sysadmins should work with the devs to get it documented.
> I guess I still think there's something to be said for the cathedral pace, though.
I think it depends a lot on your resources as a department/company. You should always execute as quickly as feasible given your team size and work load. Otherwise technical debt has a way of piling up faster than you can offload it.
At my organization, leftpad is not a reason for SRE to tell developers they can't use dependencies. Instead, leftpad is a reason for SRE to run internal package mirrors for all our supported packaging systems (debian, pip, glide, Maven, etc) and ship forks of the build tools so that when you reference a 3rd party dependency, the URL is rewritten to one at our internal mirror. The internal mirror, in turn, goes out and downloads anything it doesn't already have.
They also maintain the base docker images that we're expected to use, as well as the docker build infrastructure.
I have some perspective as someone who has done a bit of all of these jobs over the last 10 years as well as working in a hosting company that handled release management for large Fortune 500s.
> So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK?
Proprietary software generally has a support agreement and SLA for fixing things instead of getting the response "it works in dev!"
> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.
That's why containers are great, but you have to remember most of the world isn't as fast as this community to adopt things, a lot of things are still being managed the hard way on shared servers with literally thousands of dependencies. Migrating to containers in these instances can't happen fast enough.
> There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.
Implementing even trivial changes to processes that impact hundreds of people across multiple continents is often not trivial. Devs in India, devs in the US, hosting teams, release management, etc. A lot of those people are doing just enough to get by and not up-to-date tech wise, so not only are you implementing new tools and processes, but you're building out training programs around using them, etc.
These processes are old and will be modernized in time but that's the reality for a lot of "sysadmins."
>> Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer).
> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.
This illustrates why developers should have some experience with administering
systems: do not deploy unrelated services on the same machine.
And you know what happens as a byproduct of this rule of hygiene? Suddenly
the version conflicts disappear, at least for things that aren't broken
anyway.
> why should a sysadmin who doesn't even know Python
Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engineer") for a Python codebase that doesn't know Python, well, that's a different question altogether.
> I have regression tests to catch if an upgrade breaks anyyhing
You don't know what you don't know.
When you can reason about the multiple ways the above statement can fail, congratulations! You are now a seasoned sysadmin, the scorn of junior developers who just want to get things done (who incidentally read a great blog post the other day about a new packaging system that we should immediately transition to and by the way it's all backwards compatible).
That's the point of regression tests. The sysadmin also doesn't know. Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.
Look, do you want a gatekeeper or not? For your small little web project you don't need it, and a little downtime probably isn't catastrophic. But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager, at which point you hopefully have other specialized roles for security as other things as well.
I understand this does not make sense when you are not more people than can fit around a table, but as you grow you will feel the need for more and more specialized roles to fit the changing requirements. The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas, as they rightfully should be, you're going to have a problem.
Where gatekeepers are required (because regression testing is not yet fully trusted enough for continuous delivery), QA should be the gatekeeper, not sysadmins.
>For your small little web project
My comments are based upon working on projects with a turnover of > ~1-1.5 million USD / day.
>But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager
Every time I've worked with somebody whose role was "change manager" this role was introduced:
* As a response to repeated downtime in the past caused by some kind of idiocy.
* They were required to "sign off" on releases purely as an added bureaucratic step to cover some manager's ass.
* They never once prevented or caught a production issue.
* They always slowed down releases.
>The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas
Ironically the whole idea behind devops (which I fully agree with) is that it should not be a specialized role - developers and ops teams should be blended.
This is precisely because if the two teams are separate and one throws code over the wall to the other then things will go wrong. Then a manager will insist on a gatekeeper.
I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides.
In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin things just like the devs, so that they can bring their expertise to the container, rather than having to fight it.
---
Mind you, my opinion might not scale - I'm part of a small enough team that I'm holding both those roles, but I try make sure to spent time wearing both "hats", so that one role doesn't get more man-hours clocked.
Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete).
Minimize your dependencies. It's incidentally also what leads to clean code bases.
>Minimize your dependencies. It's incidentally also what leads to clean code bases.
Oh hell no. I have wasted far too much of my life maintaining buggy, technical debt ridden reinvented wheels where there was a well maintained package that could just have been used instead.
> Minimize your dependencies. It's incidentally also what leads to clean code bases.
You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?
Are you arguing that you can keep velocity while building up technical debt?
The company "smartly" using libraries might stuck maintaining a monster of dependencies that only ever was meant to be an MVP. It will require 10x more engineers and while they might move fast at the beginning, they will only slow down over time.
The company minimizing their dependencies and paying attention to their stack will be able to add complexity over time without breaking a sweat. Their costs will be 10th the cost, and they will be able to run profitably.
I am not anti-library or anything, but e.g. adding sci-py to your python project just because you need a gaussian function in one place in your code is just lazy.
Managing dependencies wisely is one of the hardest thing in software development.
It's right after cache invalidation and naming things ;)
In my experience, the vast number of problems we've had with our mobile apps, especially on Android, has been a developer deciding to use some random SDK to solve a simple problem because he didn't want to take the time to write it himself.
Then the app is broken or has memory crashes, or the final binary is 10x the size it needs to be. 9 out of 10 times it's a third party library. This is why I ban the use of them unless absolutely needed.
To call out something that nobody else mentioned: because when you to use that newer version of pycurl or pyASN1 it's probably going to break the system level tools for running patches, handling license enforcement, and being able to keep the database online for other teams.
I've had this problem over and over again (biggest one last being with the US Census). Folks insisted on upgrading a python library and auth to the hosts stopped working.
> 15 years ago, when a project would kick-off, as a sysadmin
> I'd be invited in and the developers and I would hash out
> what versions of each language and library involved the
> project would use
To wit: the new ideologies in infrastructure management are actually designed to solve the underlying problem that necessitated that kind of working setup. Why should the version of a lib in one part of the software somehow pose existential threat to the infrastructure? Engrain the dependencies into contained, independently deployable pieces, and make it so that app-level code can evolve without bringing down the world with it. Make it easy to revert back, and/or utilize phased rollouts, and you've got the ability to iterate quickly, keep pace with external dependencies, and it no longer has to be some scary thing that requires big back-and-forth meetings over mundane details.
(As for software that releases often, maybe it's an over-correction, but there's a reason things don't work as they did in the glory days, and that's because they were never really that glorious.)
This doesn't necessarily rule out the expertise of systems administration, because the platforms for all of this need to be built & maintained, and there's still a lot of work to be done on network boarder security, etc. It's a movement that focuses systems administration to systems administration, instead of having to be this big org arbiter of microdecisions, and all the baggage that goes along with trying to be the gatekeeper of all.
> Why should the version of a lib in one part of the software somehow pose existential threat to the infrastructure?
Because that's how software developers wrote every dominant packaging system :P
There are tradeoffs to self-contained units. Disk space isn't so much of a practical concern these days, but security is very real: with a dozen apps, you could be at the mercy of a dozen different entities to update their embedded OpenSSL libraries.
this is the core of the problem between Devs and sysadmins. Sysadmins come from a mindset of a polished working system which never needs to change. They deliver stability and reliability to the business.
Devs come from a mindset to actively create change. This is to add new features and deliver new value and product to the business. As a Dev I do have to say that many Devs don't have enough experience in operations to understand properly how to help sysadmins, many don't understand the complexities of that job.
These two perspectives are at odds, and they should be. The new tools, like docker, start giving everyone what they want... Devs pick their dependencies, and in theory, can't stomp on the sysadmins pristine environment.
To respond directly to your question: because there are new things available in new libraries that allow us to develop new features!
The new things you need to develop new features are far and far between
99% of web software written these days could fulfil identical use cases on an IBM 3270 from 40 years ago. You enter something into a form and it gets stored in a database. You enter something into a field and it generates a report. That's all Amazon, Facebook, Google, any e-commerce site are.
Sure it might be nice to use a new version of that new JS framework that all the twitterati are going crazy about, but does it deliver value to the business that justifies the risk and investment?
And yet none of those things did arise 40 years ago. All of the nuances of all the code written since then make a difference, despite duplicating "identical use cases".
People could do great things just with punch cards, yet somehow technology kept marching on.
If developers want to use newer stuff usually they have a good reason. The ability to hack around the deficiencies of old dependencies does not mean that one couldn't get a better, cheaper solution with newer technology.
> People could do great things just with punch cards, yet somehow technology kept marching on.
That's not the situation I've described - punch cards disqualify.
The situation I mean is where developers insist on writing software on version X, which doesn't compile on X-1 and is buggy on version X (and might not compile again on X+1). For a concrete example, new C++ features that aren't correctly implemented and lead to harder to read code and worse error messages when applied to day-to-day problems (which these features were never meant for).
If it is as you say, then why upgrade ever? How would we even discover bugs in software until it is used?
To have progress we need to change things. When we change things, we may break things, regardless of tests.
To quote deijkstra: "testing can be a very effective way of showing the presence of bugs, but it is hopelessly inadequate to show their absence". From 'the humble programmer'.
Production is the only way to eventually discover the stability of any software, even with 100% test coverage. It's a necessary evil in the support of progress.
If you run into a bug or problem with a 3rd party component (open source library, commercial tool, whatever), one of the first things they are going to ask you to do is upgrade. The fact you're on an old version of some library is an easy (and sometimes correct) scapegoat for problems.
Put yourself in the 3rd party's shoes: if you spend a bunch of time trying to fix a problem that turns out to be a bug in a separate library that's already been fixed, that's entirely wasted time.
The same goes for direct usage: you're likely to spend time fixing problems that have already been fixed.
Upgrading the version of the library wouldn't be a problem if the concept of stable ABI's were as prevalent as it was 15 years ago. Back then, the major.minor version number system was used as a signal that it was safe to upgrade to a newer version of a library without worrying that the entire application stack was going to come falling down around itself because the developer of said library decided to rework some part of the library without providing any backwards compatibility.
Put another way, a sysadmin could feel confident that moving from 1.52->1.53 would be a painless and transparent operation and that the provider of said library would continue to release 1.x branches with little ABI changes for some length of time. The expectation was that at some point the library provider would release a 2.0, which would require a more careful testing/deployment schedule likely with other upgrades to the system.
Today, that is all out the window, very few open source projects (and its infecting the commercial software too) provide "stable" branches. The agile, throw out the latest untested version mentality is less work than the careful plan/code/test/release, followed by fix/test/release, cycles.
This is a major rant of mine, as upgrading the vast majority of open source libraries usually just replaces one set of problems with another. Having been on the hook for providing a rock solid stable environment for critical infrastructure (think emergency services, banks, power plants, etc) I came to the conclusion that for many libraries/tools you had better be prepared to fix and backport bug fixes yourself unless you were solely relying on only libraries shipped in something like RHEL/SLES (and even then if you wanted it fixed fast, you had better be prepared to duplicate/debug the problem yourself).
> Put another way, a sysadmin could feel confident that moving from 1.52->1.53 would be a painless and transparent operation
This is what Semantic Versioning [1] aims to achieve, but as you highlighted, it still requires the maintainer(s) of the project to actually deliver stable software, regardless of what the version is. I think some people took "move fast and break things" a bit too literally.
A project following SemVer and that has good automated test coverage is definitely on the right track though, and in generally should be a pretty safe upgrade (of course it's important to know their track record).
"Move fast and break things ... in a separate branch with continuous integration running an extensive test suite" isn't quite as catchy but is what should be happening.
> The same goes for direct usage: you're likely to spend time fixing problems that have already been fixed.
That depends on whether it's a feature or a fix release. Feature releases might or might not include bug fixes, but they typically include new bugs. I welcome localized fixes, however they are not as common because of constrained resources. (Fix releases is the idea behind Debian stable. Of course it only works to an extent).
A different perspective, I prefer to have the bugs that I already know, and know not to trigger.
Because those libraries have bugs, sometimes catastrophic ones. Sometimes they must update, due to API changes or other factors outside of their control. If your organization relies on keeping things static as a means to stability, one day that rule will have to break, and you may be pretty underprepared for it.
The reason why virtualenv exists is because different apps may have conflicting requirements, and you have apps that need to be deployed in different environments with different versions of different libraries. I know that even if I were developing against versions of libraries in system packages, I'd still end up having to use virtualenv in development (EDIT: I wrote 'production' here by accident) because my stuff gets deployed on different versions of Debian and RHEL, necessitating virtual environments if only so that I can make my development environment as close to production as possible.
> In the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities.
Much of that has been down to efforts in recent years to finally fix the major issues with Python packaging. It has settled down quite a bit. Also, the 1.* to 8.* change is because the initial '1' was dropped: 8.* is essentially 1.8.* in the old versioning scheme.
I'm not saying that this couldn't have been handled better, but it's not just a 'hummingbirds off of their ritalin' situation: Python spent many years with packaging stagnated, and what you're seeing is rapid development to fix the mess that years of PJE-related neglect caused.
> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project".
As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask.
This stuff tends to get done years after the original devs have all moved on. Maybe they tried that kind of thing back in the early days, before I started working with Ruby, definitely not today.
The time I'm thinking of was a team that wanted to switch from Rails 1.2 to 2.0 along with whichever interpreter bump was required to make that happen (IIRC 1.8.5 to 1.8.6, but this was a decade ago; I'm pretty sure 1.9 hadn't come out yet). Halfway through a project.
DevOps here - yes, they do bother. Pinned set of dependencies, but one of them updates? Upgrade all of the dependencies. But don't worry, it's all in a Docker container (I have a completely separate rant about Docker's compatibility ignorant hummingbird).
Ironically enough, I think the current DevOps culture emerged partially because sysadmins got tired of saying no (if only so they could sleep through the night), so now they let developers tie their own nooses so they can be woken up at night.
It's wonderful to give up all of those software pages back to developers. And the developers do seem motivated to fix the bugs which wake them up at 3am, so it turns into a win all around. It's still hard to watch a new team come up to speed though, knowing how little sleep they will be getting over the next month because they made their new docker program stateful...
Ironically all-aboard-the-update-train was the actual reason I jumped off IIS back in the 90s when I got badly burned by updating Windows NT. Automatic database connection pooling for IIS was dropped and I started getting annoyed phone calls from clients who's websites were dying after updates.
One had to read MSDN every day to keep up with what might break on sites you had no control over.
The problem is that sysadmins don't know everything.
> in the year 2015, pip went from version 1.5.6 to 8.1.1
The only releases in 2015 were 6.x and 7.x.
There were 8 documented backwards incompatibilities, 4 deprecated the previous year, and 3 documenting a couple bugs that were fixed several days after the 7.0.0 release.
These are the sorts of thing an aware Python developer will know.
You're right; it was the period from December 22nd, 2014 to March 17th, 2016, so about 15 months centered around 2015.
We may be counting regressions differently; I'm including both adding and removing the spinner as a regression, for instance (since both the addition and removal added unexpected behavior).
Note that the undeniable regressions that occurred in releases during those 15 months included:
1. Exceptions raised in any command on Windows
2. Switching from not installing standard libraries to installing them back to not installing them
3. Blocking if the particular server pypi.python.org was down
4. An infinite loop on filesystems that do not allow hard links
Note that in that time they also added yet another internal package management system (incompatible with the existing two), changed the versioning semantics twice, and dropped support for versions of python that were 3 years old at that point.
And, again, there's nothing particularly wrong with or bad about pip; this is just what a younger generation of developers are used to.
> You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases.
Releasing an RC often results in nobody using it and hence not finding the bug even in several weeks, but it gets caught almost instantly in a release… At least, that's my experience in shipping various RCs that have led to next-day regression-fixes once it does ship.
While yes, better testing would solve such issues, but at some point the line has to be drawn as "good enough", because there's ultimately a limit to what is reasonable.
Oh, definitely, and I want to be clear I'm not wagging my finger at pip here; it's a good project. It's just that for somebody like me on the far side of 40 (and sysadmins in general are a grayer cohort than devs), that's absolutely not a release tempo that I grew up with. I still shudder when I see a Dockerfile that begins "FROM whatever/latest..." because I have no idea if whatever/latest is going to be the next leftpad.
That might be a bit naive, but isn't that more of a package management Problem than everything else? I can tell you that while I know this issue I also know that on systems like Alpine Linux or FreeBSD this is less of a problem than it used to be and containers (so virtualization - even when it is OS-level) is potentially overkill and certainly not a solution. virtualenv and others seem way saner in this case.
For development as a whole it is really great though in my opinion.
Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility?
The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?
That's a good question. I think it's not a coincidence that CPAN predated the widespread use of distributed source control systems whereas pypi and gems blew up just as mercurial and git were unseating svn and cvs. It's a different release philosophy (remember, in the 1990s you often didn't even get to see pre-release CVS commits of open source projects; that was an innovation of OpenBSD).
I also think the widespread use of VPSs rather than accounts on shared servers (again, containerization) was a factor. In the 90s and early 2000s, you usually (even in a corporate setting) had an unprivileged account on a server with a given version of apache and perl, your own cgi-bin directory, and possibly some latitude on a personal CPAN install directory. The lack of containerization meant you had to compromise between using newer software and breaking existing use cases.
So I guess I think it's not so much about Python vs. Perl per se but about the technologies available when those languages became popular among developers.
That would seem to be ironic. As a longtime Perl developer who switched (for pragmatic reasons -- a job) two and a half years ago, my impression is that Python [as a language] is much better suited to a business environment. What makes Perl such a wonderful language and why I enjoy it so much is exactly why it blows as a business language. Python's rigidity is very useful if you ever want someone else to read and understand code written by someone else. So the idea that Perl ends up with the more mature package management is exactly the opposite of what I would have predicted.
That said, I haven't had any more problems with PyPi packages than I did in the past with CPAN. Yes, pip always wants to upgrade itself, but that sort of every-damn-day software upgrade cycle seems to have become quite prevalent, not just in the Python world.
You talk about that Python have just one possible layout standard and so on, I guess? That is a different subject. (And sure, no coding standard is bad for a project, Python removes that discussion to a degree.)
I think the real problem with Python/Ruby/etc is the surprising lack of an analogue to CPAN Testers.
It isn't just all of CPAN that is tested on different OS/Perl version combinations, it also stress test the Perl versions.
Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason.
Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can't be stopped from having the newest version of things, and I think that's a bad idea.
15 years ago, when a project would kick-off, as a sysadmin I'd be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure.
Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I'm not calling pip out as particularly bad; I'm calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Ruby is just as bad; the famous (and I've experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it.
Devs are chasing a moving target, and devs who haven't been sysadmins may have trouble seeing why that's a bad idea.
[1]: https://pip.pypa.io/en/stable/news/