This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin.
Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.
And I've never thrown code over a wall without having tested it in a representative environment.
The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone.
When a sysadmin sees an issue in "their" environment that they understand there's a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it's the most trivial issue the developer has had to resolve that day.
Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them.
Sorry. There must be great sysadmins out there struggling with terrible developers, I'm sure of it. I just haven't seen it.
I've done the dual sysadmin/developer thing for a small company, and the problem I experienced there was completely incompatible working modes.
Sysadmins must deal with interrupts (requests, crises, things driven by external schedules etc) and then in the rest of their time build systems to manage or reduce the interrupts. Developers are expected to produce work on a predictable schedule. This is disrupted by interrupts and obliterates the schedule for proactive work unless your management is very good at making it a priority.
The "prevention of information services" problem is certainly real though. Perhaps it could be addressed by embedding the sysadmins in the dev teams rather than having a department of their own, but then you have to fight org hierarchy.
I had exactly the same experience at a previous employer, almost word-for-word.
Having said that, the central assertion is still correct: the absolute best developers I've ever worked with were also top-tier sysadmins (or linux experts, depending on what you want to call it).
AMEN!!! In my current job, I am wearing both hats, and while I like that there is a certain variety in my work, users calling for help is highly disruptive when programming or doing some other stuff that requires deep focus.
The upside that in a three-person IT department there is very little bureaucracy to fight, just the odd "organically grown" legacy system.
As an aside, did you know that the word "Amen" actually is a acronym in the Jewish language that means "El melekh neʾeman" (or AMN) which translates to "God, trustworthy King". (source: https://en.wikipedia.org/wiki/Amen)
I figured the etymology of that word was rather interesting. But yeah, I get the whole SysAd/Dev dual job. They're tough to balance and do effectively. SysAds are firefighters. When the nag(ios) alarm rings, we come a-callin.
> As an aside, did you know that the word "Amen" actually is a acronym in the Jewish language that means "El melekh neʾeman" (or AMN) which translates to "God, trustworthy King". (source: https://en.wikipedia.org/wiki/Amen)
The fact is that nobody really knows. Talmud was written in ~200 BC and is an exegesis. Egypt is the elephant in the room of Hebraic history. It is possible it is an Egyptian loan word just like Moses -- "born" from water -- is an Egyptian name.
I've been in this situation for a long time; The worst part is every so often you get assigned a PM who wants you to accept responsibility for meeting artificial development deadlines.
Developers IMO benefit greatly from having the general engineering experience. This helps understand how the part they build fits in a full product, where the narrow spots are, what is likely to break first, where formal documentation is insufficient / contradictory / wrong, etc.
Sysadmins, who often manage crises, acquire this experience way or another (e.g, by researching options to fit a square peg in a round hole without leaks), so developers with sysadmin experience tend to all have it. I think though that the key part is the "engineer" part and it can be acquired and used without sysadmin-imposed hassles (interrupts, crises, being underappreciated).
> I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going
I'm sorry you've had such an awful experience with sysadmin colleagues that you've developed such a corrosive attitude towards them. I've worked in lots of good environments, where dev/ops was being effectively practiced, and sysadmins there were the most effective force multipliers imaginable.
> awful experience with sysadmin colleagues that you've developed such a corrosive attitude
This is going to be a sensitive topic, but can we talk about ""BOFH"" culture somewhere on this thread? (Maybe I'm old and it's now dead, but I think some of it persists)
I kind of understand it as a product of working in an environment where everything is urgent and nothing is appreciated, but when sysadmins come to resent the people they're supposed to be supporting then the force multiplier turns negative. Sysadmins develop strategies for reducing the number of requests at any cost, usually by making the experience as opaque and unhelpful as possible.
When I was reading BOfH a few years ago I got a different gist.
The BOfH is the archetype of someone who is excellent both with technology and politics. When you are in a service role, you have two competing priorities. You must deliver people the things they want, but also keep things nice and stable for yourself so you don't go crazy.
An important dynamic in organizations is laziness vs. intimidation. Political savviness allows you to apply intimidation to get the lazy to do what you want. You can threaten to fire, or raise an issue that could possibly get them fired and even if it doesn't, won't make you look good. The BOfH is someone who can respond to political intimidation with adroit technical interventions to ensure that his second priority, ensuring a smoothly-running system, isn't threatened.
If you read the BOfH stories carefully, you see that the operator knows where his bread is buttered and is careful to remain on good terms with the people who really have the power in the company. The whole thing is a phenomenal read on organizational dynamics.
> Sysadmins develop strategies for reducing the number of requests at any cost, usually by making the experience as opaque and unhelpful as possible.
This is sometimes an organizational problem. I worked in a support role at VMware for about 5 years and this is what I observed:
- Support & IT departments typically have enough staff in the beginning
- The organization grows & the department grows to match the new work that exists
- At a certain point, the organizational view of Support & IT/Ops changes, and it's now viewed as a cost that you want to keep down.
- Leaders try to minimize the increases in budget, but the workload per sysadmin/engineer increases.
- The sysadmins/engineers have no control over the flow of new work, which effects the quality of work that gets done and can create a toxic environment.
It literally becomes impossible to handle all the incoming requests. Different people handle it differently. Good sysadmins would learn to prioritize properly, but due to the toxicity some people have trouble handling it so they end up developing strategies to make a certain number of requests "go away".
1) Leadership: Stop viewing the IT/Ops/Support department as a "cost to keep down".
2) Leadership: Treat the department like they are manned by people.
3) Realize that not all requests are created equal. Some take minutes, some take months.
4) Determine a reasonable number of requests/tickets per sysadmin/engineer. Make sure to add padding for things like project work, sick time, vacation, professional development, and so on.
5) Hire proactively to prevent the determined threshold above from being surpassed.
6) From the IT/Ops departments perspective: realize that the incoming requests are coming from people that need your help and they are effectively your clients/customers. Treat them as if customer satisfaction is extremely important!
There are also other strategies where you give a subset of people the ability to work on projects and designate a different subset to be interupted with urgent requests, and rotate the role. There are all kinds of things you can do to improve the situation :)
Sysadmins are, 99% of the time, responsible for production uptime. Sadly, management includes all of the software being written in-house in this expectation. Change means instability, means that Sysadmins' feet are held to the fire - this makes them resistant to change.
Software developers, on the other hand, are responsible for making changes. Adding features, pushing fixes, and so forth.
These two points of view are inevitably going to cause friction. Developers are only recently starting to be held responsible for production uptime and the pages that come along with that - and it's a good thing for both sides.
That "most trivial issue" for a developer is something the sysadmin was woken up for 3x in the past, and doesn't want to be woken up for again, so he pushes back. How can he not?
In 1997 when my code first started running in Live I was given a pager and told welcome to Ops. Every piece of software that company had a single dev name easily read out from the binary and used to contact when trouble occurred.
>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour.
How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?
> How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?
Having worked in ops for > 10 years, this is how it usually goes.
The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them.
Management always pushes back on this tactic. That's reasonable; they need to justify capital expenses (especially if you're self-hosted).
The core issue though is that capital expenses are easy to quantify, while "lost productivity" is much harder to fully account for. If I complain that some hardware upgrade which costs $x could improve productivity, I just don't have hard numbers on my end - it's all napkin math.
In many places reluctance to spend money on infrastructure is also, I think, a symptom of headcount-itis. Managers love to have more employees, and love to have more for them to do, because that makes managers seem more impressive to the org. My manager might have perverse incentives; keeping the SAs busy fighting scaling fires both makes his team look impressive because they're busier, and makes him look better because the capital expenditures are lower.
Obviously, head count is expensive, so this is usually a game of appearances rather than an effective strategy to improve the bottom line. Good insight into productivity is required to catch this kind of stuff, but in the real world I've found that a lot of places just don't have an org structure capable of weighing cost / benefit properly when it comes to infrastructure.
How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?
If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is that having?", and call a wider meeting that brings together the higher-up-the-foodchain manager, the development team, and anyone else who might be affected. Ideally it'd be the higher up manager who calls that meeting of course, but they may not understand the technical issues.
I have long thought that one of the most effective workplace sabotages a sysadmin could put into place was to implement management plans without question.
>If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job.
This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along.
Please don't fall on the sword for incompetent managers.
This is the responsibility of someone above to know whether or not the orders they give should be given.
Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.
Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side.
All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!
My favorite is being told I can't store stuff I need on some enterprise storage solution because it is running out of space, when I know in my head a couple more terabytes of storage costs way less than the amount of effort that went into discussing it by all parties involved, and there's no effort to help me find an alternative. So it goes into a different bucket, like S3, which is what we were trying to avoid (for various on-premise benefits) in the first place.
I agree. I also think it should work both ways. The worst jobs I've ever had is when the sysadmins have the mindset where they own the servers and are unwilling to deviate from what they've read at the behest of the developers.
"I will force AV on reads on the developer boxes."
"I will install AV on the production DB servers without telling anyone in the development group, then make the developers prove AV was the cause of production slowness before removing it two weeks later."
"I will force this crazy group policy on developers and when they complain, I will totally ignore them."
A bad, or uncompromising sysadmin (one in the same) make development work a complete nightmare.
I half believe the reason developers are embracing cloud architecture so much is to remove so many sysadmins out of the equation.
On a side note, a tip for developers. Always make friends with the sysadmins. Buy them lunch or something. Right or wrong, they can make your lives much better or much more miserable.
> I half believe the reason developers are embracing cloud architecture so much is to remove so many sysadmins out of the equation.
It was literally true at one of my previous jobs. We couldn't install anything on our own dev machines without approval from Net Ops, not even Notepad++ (I don't think I ever got that installed, never got approval).
We once asked for a new server which mirrored the software of an existing server with two months lead time and got complaints that two months is not enough time to get a new server. I think we ended up getting it in three months, after the new project was supposed to be deployed to it.
Meanwhile we were starting to get into Azure, and we had a new server in Azure up and spinning with everything we needed installed on it in about 15 minutes.
The Lead Developer said, "We need to get as much stuff on the cloud as we can so we can stop dealing with this mess." We dealt with a lot of PHI there, though, so there was only so much we could do.
I've seen this as well. "Timeline from internal IT for provisioning a box and deploying our app is 6 months, and subsequent changes go through a ticketing system with a 2 week average turnaround. Or, we can have it running ourselves on AWS in 30 minutes."
Shit. So, you're saying that my mess of a team is kinda awesome by implementing reliable production-ready deployments within 2-4 weeks, and implementing changes to environments within like 30 minutes ("set key foo to bar in configs please?") to a week ("we need persistence!").
I guess IT in this place really is getting up to speed.
what your probably not seeing, is the CIO/CSO screaming at the SA to get AV deployed on every machine in the company, to meet some audit requirement checkbox, or PCI compliance, by the end of the month.
Exactly this. Audits don't care if there isn't any practical malware or if nobody can access the system outside 3306 and 22. Audits say "all production systems implement antivirus software" as a binary checkbox.
When I have seen this problem, it's because the sysadmins are instructed (or have learned via experience) not to explain their reasoning to developers or end-users. Because if they did, then it becomes a discussion or argument that becomes a time sink since there was very little chance they could change the mandate even if they agreed.
So they become intentionally opaque to move that discussion out of their laps and make it come via the development team managers confronting the operations managers and having the fight on that turf.
Such situations occurring is a sign that the organization is not set up effectively. This sort of confrontation shouldn't need to be happening.
Ideally the development team's lead and/or project managers are involved with, are informed ahead of time, or are even contributing to the policy decisions on the operational side.
Because telling someone that you did something because of compliance doesn't help. They still blame you personally even though the compliance standards are usually industry-wide or even defined by Congress as an act of law.
One of our web teams wanted to do a simple Wordpress deployment on LAMP. As sysadmins, that was no issue, even with clustered mariadb. But our DBA team doesn't have any experience outside of Oracle/MSSQL, and squawked about mariadb. After an hour of this BS, the manager of the web team spun up a few EC2 instances and got to work. Of course we don't have anyone familiar with EC2, so supporting that will be a learning curve for someone, but the manager is happy, he has his own sandbox without hassles from the DBAs.
"This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin."
Couldn't this argument be applied to any developer for any discipline/speciality?
Sure, more knowledge/context is always better if reasonable to attain, but my experience suggests that your above concerns could also be addressed via team organization rather than expecting all developers to know all things.
Not particularly controversial at all, from my POV.
I was a sysadmin with various ISPs in various countries for 15 years before I "turned to the dark side". I'd been using Ruby for a few years with Puppet and Chef, and after dealing with one too many "flaky coders", I picked it up.
I have to say, coding is far more enjoyable, though both come in handy in my day-to-day life.
It sounds like you've dealt with a few "BOFH" sysadmins. Don't worry, we're not all like that, and those that have been on both sides of the team will probably see your way.
Tell your boss I'm available (remotely), by the way ;)
I have seen this situation too many times (exaggerated a bit):
D: I have noticed that task Frobnicate has not been running in Production for a month, then checked and it is not even added to scheduler!
SA: There is no mention of Frobnicate in the pipeline for scheduled tasks.
D: What pipeline? FancyPancyScheduler is bundled with application and tasks are defined in DB, I have done it in Staging and everything worked, Frobnicate is all the fuss in the team, you must have heard about it, why don't you check for changes in Staging?
SA: We have well defined pipeline to manage scheduled tasks, currently the executing agent is Cron, not FancyPancyScheduler.
---------------------
Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure). In my short career I have seen problems arise where one party makes plans and changes according to current or even past (it worked like this earlier) state of the other party. This applies to both developers and admins.
So I sort of agree with your sentiment, that developers need understanding of system administration. Though, depending on team size, I believe it is entirely sufficient to have someone in Developement who understands system administration and actual infrastructure, and someone in Operations who understands developement and actual stack. This is where I hope DevOps will end up at: arbitration between Developement and Operations to ensure smooth sailing forwards. Because the debate "I will do it in code" versus "this must be done on the edge" (e.g. static assets in a website. Served by application or web frontend?) will never be resolved.
> Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure).
I disagree because this generalizes both developers and admins too much for my own comfort. I've seen sysadmins get really sloppy in the name of getting something into production quickly out of hubris without thinking about the full lifecycle of an application (common with developer-turned-sysadmin engineers - I am one and tend to be more reckless due to the reality that most of the errors I've observed would not have been caught going super slow - that adding more test code does not necessarily find the most critical of errors, just increases confidence) and especially in enterprise software most developers are sitting on features and are nearly allergic to new trends by their organizations valuing revenue loss far above losing growth opportunities.
Of course the stereotype is that operations wants things stable and manageable at the behest of business while developers want to deploy new stuff faster (because the idea of development in most places is to create something new). Modern infrastructure becomes increasingly code-driven and emergent as opposed to manually formed and restrictively managed sysadmins will have more room for errors that may change this into the future. Meanwhile, developers are increasingly under greater scrutiny by society when rolling out features such that nobody can ignore the concerns and they may be eventually forced into nearly waterfall-like development patterns. We can already observe this with the infection of Agile with enterprise bureaucracy / overmanagement back into the rest of the software industry as many of the former smaller, agile tech companies become big behemoths themselves.
DevOps approach definitely shifts some sysadmin-type roles towards the development teams.
That said though there are things that belong in the realm of sysadmin responsibilities - both "old school" like let's say setting up DB replication, VPNs or a puppet master server(not that "old school" but still) and "new" - things like Kubernetes and Fleet/CoreOS for example have plenty of configuration/maintenance complexity that is better suited to be handled by a dedicated sysadmin.
> thrown code over a wall without having tested it
this is the weirdest part of the whole devops mantra. like, I know how to evaluate the complexity and memory requirement of code way before I write it and I guess most compsci should be able to do the same.
so either it's yet one more attempt in getting cheap labor into workable territory or plenty people where this myth originated are being cheated out of their money for a graduated curriculum that teaches nothing of value.
Those are only tiny slices of real production bugs. No amount of complexity analysis of your code ahead of time is going to protect you from all of the issues that arise with integrating any large system dealing with lots of requests. You run into all kinds of things like query optimization, kernel TCP tuning, load balancer problems, cache thrashing, high latency clients, out of spec clients, power failures, etc.
If you think knowing the theoretical behavior of your program in an ideal environment is enough, you are exactly the type that throws code over a wall without having tested it.
funny how most of the things you list are either stuff that can be audited in code alone (query optimization, cache thrashing) or totally out of control of the developer (load balancer issues, tcp tuning)
sure if you bounce them all up like that it might look like you have a point, except it falls apart when you attribute concerns properly.
or please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code?
Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query?
Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going to be deployed on. It's highly unlikely that the servers will have the same processor cache sizes, memory sizes, and numa architecture of the dev's laptop.
Load balancer is something a developer should know about as well. A developer has to consider the behavior required by the application (e.g. backend session persistence, headers injected, etc).
>please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code?
Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections for the app to function. He might also learn that TCP isn't very efficient on high bandwidth, high latency, lossy networks and decide to switch to UDP with error correction.
Long story short, a good developer should know everything about the environment in which the app is intended to run. "It performed ideally on my laptop" is throwing code over the wall.
Civil engineers don't design bridges without understanding where the bridge will go. The same applies to software.
> Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections
so we're back to point one, you need devs that go trough basic education and stop cheapening out hiring Joe / or Joe should ask a refund from his tuition fees.
> snip of stuff that one does not know off the bat
sure but it is knowable, it's not exactly hard. database are predictable, building indexes on the right places is not an esoteric practice that can only be done by trial error and rituals etc etc. literature is quite adbuntant, easy to process and complete with tradeoffs about different approaches and how they impact performance, maintainability etc.
>so we're back to point one, you need devs that go trough basic education
There is no basic education that covers the associated costs of a TCP connection in the kernel of a modern operating system or in the load balancers it passes through on the edge of the network.
>sure but it is knowable
So you're saying it is important for a developer to understand the infrastructure the code will run on. Thank you
The reason I brought up all of those points is because they are things not covered in CS educations and they hammer "hands off" devs all of the time.
I've worked with tons of junior devs from all kinds of good schools (Stanford, MIT, UC Berkeley, etc) and they almost always get bitten by this stuff because they throw their code over the wall and don't make an effort to understand the operational environment. It has nothing to do with a good education, it has to do with a mindset of not operating in a vacuum.
Many devs out there who work with Windows or do mostly front end often have little experience in that domain.
Seeing alot of work get done at uni by students - who also actually some backend (friend did a blockchain project recently) did infact do very little backend discovery - the job was delegated to another student to get the env. Up and running.
> Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.
I've had the exact opposite experience. In most of the organizations I've worked in the "sysadmins" (mostly Systems Engineers/Operations Engineers actually) were stronger developers than the people who were actually developing the software. But that could just be a title shift, because what I've seen happen is that people who care about systems but have a development background gravitate towards operations roles and end up filling in as the "actually Senior" developer for the dev teams.
In the 15 years I've been doing this, I've only occasionally met someone who has stuck hard to the development side of the house but actually is competent when it comes to systems. Most developers have zero care about any of the lower level things like networks, hardware, and even backend software/databases which are required for their application to succeed. A common scenario is that the devs choose an inappropriate backend stack because they chose the easiest things to deploy rather than what is best suited for the use case. Then when things blow up, they beg for an ops team to be created, which usually starts by hiring people who are competent enough developers they can relatively painlessly replace the entire backend with something sane (e.g. Mongo to Postgres shifts are commonplace, because Mongo is a dog in the real world).
> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone.
Yeah, that does sound terrible. I agree. My top 5 jobs as a systems person is the following in priority order:
1. Make sure production stays up for our customers so we keep making money. (5 9s targets)
2. Ensure the security (and compliance) of our systems so we don't get hacked and we maintain customer expectations about compliance.
3. Ensure the performance of our product/systems is up to customer expectations.
4. Make sure deployment automation is solid and streamlined so that deployments are frictionless
5. Make sure new code is actually being deployed regularly and remove impediments to deployment so customers get features faster.
You'll notice a trend here I'm sure. The most important thing is the customer, then the developer. The biggest frictions I've seen between systems/development teams is when the development team believes that their desires/needs are the highest priority. The systems team is /not/ there to be at the beck and call of the development team, it's to be at the beck and call of the customer who is paying the company money. As much as possible I try to ensure the development team is having a frictionless experience, but if something will negatively impact the customer it is 100% my job to throw a roadblock in the way of the development team to prevent that. The customer of the company is my priority, and everything else is secondary.
I believe in small cross functional teams but that you need to be a sysadmin to develop I don't agree with. Perhaps it's more your opinion of what a good developer means, most teams benefit from variety in my experience. It sounds like your biased to certain types of organisations where there's a big gap between departments.
There's no value in arguing over the semantics of "engineering". There are huge differences between software development and e.g. civil engineering, to the point that I would be dubious about any analogy that treated them as the same thing.
Chemical engineering, Hydroelectrical engineering, Power engineering come immediately to mind as engineering disciplines that deal with active systems that require operational management and control.
Of course. (In my opinion, /high software/ has more in common with mathematics, music, theatre-film-dance, and architecture than it has with engineering, and /low software/ is begining to resemble boiler room operations.)
And to this day, we hear about "software engineers" and "software engineering".
Per my OP: "It seems either your conclusion is held to be incorrect, or, we reach the conclusion that software development is not engineering."
Possibly, one reason for the prevalent problems in the pedagogical & human resource fulfillment aspects of the field is due to a miscategorization of the field.
Process engineering (~manufacturing) and logistics (~supply-chain) are not dissimilar to modern software workflow. The basic tools (modular management of complexity, discrete processing, statistics, monitoring, redundancy in processes/providers, feedback) are equivalent. In fact, I feel like a huge part of a successful software career is learning to see the similarities in disparate fields and draw from them positive architectural benefits, while keeping other-profession-spire-dwellers properly onside/placated.
Well that is certainly correct but it should be pointed out that one can say that about most organized (psuedo-)industrial production endeavors. But it seems incorrect to posit that that is the 'defining' characteristic of software development.
> In fact, I feel like a huge part of a successful software career is learning to see the similarities in disparate fields and draw from them positive architectural benefits, while keeping other-profession-spire-dwellers properly onside/placated.
Fully agreed. In fact that has been my guiding light in my own approach to software development. To clarify my view, I think software, very much like architecture, is a polyglot yet distinct discipline. It is not engineering. It not mathematical logic. It is not process engineering. It is not logistics (provisioning). Etc. (Just like architecture is not civil engineering. It is not philosophy. It is not art. It is not environmental systems engineering. Etc. It is architecture.)
-- p.s. edit --
I would like to bolster my earlier statement that software development has more in common with architecture, theatre, film, etc., than with engineering:
I would like to propose and roughly define a notion of 'semantic gap'. A sort of soft measure of the degree to which the formally expressible definition of a 'production' falls short of permitting the realization of the 'product' without the intervention of the 'designer'.
With that definition in hand, I propose that "engineering" discipines are those creative productions that have minimized the semantic gap to a degree that permits strict divisions of labor in the production.
Where as the "arts" are those creative endeavors that are faced with an intrinsic constraint on the degree to which the semantic gap can be minimized, and, that this maximally reducible semantic gap requires subjective and/or contextual 'interpretation' of the formally expressed design.
I like your semantic gap notion, however I am less convinced that overall mutual comprehension is the issue but rather the different issues of clarity of expression of vision (at the earlier/design stage), or clarity of interface (beginning at the implementation stage).
By way of example, there are many successful artistic projects that utilized the talents of multiple artists in parallel (lots of murals and mosaics, for instance).
In larger scale computing projects, frequently the (mechanics of the) interfaces provide bigger problems than the vision statement or overall goal, whereas in artistic projects indefinable aesthetics may be the shopstopper, despite perfect comprehension and collaboration.
>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them.
This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Capability and Maturity Model (CMM) as a counterexample of where some companies go. Development is not at one remove but two from production support. There's an "operate" team between them and production environments and in a regulated environment operate doesn't have privileged access either. That'll be a third team due to separation of duties requirements.
Now imagine you're paged out to a call where your code is slow or failing and you're not even allowed to login to where the issue's happening. Fun, right?
This is why I'm absolutely loving the devops changes we're seeing now - because developers can control the environment without retaining control of it. My ideal is to apply some sensible defaults (no, you can't have all my crashdump space for your app logging; ask for more disk instead, no you can't run ghost/glibc/pooodle vulnerable versions of libraries) and otherwise let the developers spec the OS as a template or dependency for their app. It's much better for me since if I'm required to troubleshoot I know my requirements are met and otherwise the developer may do as they wish. Everyone wins and my control requirements are satisfied because remember developers are never allowed production access in regulated environments.
>Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.
I guess it depends on what you mean? The developer is in the best position to know what logging there is and how to enable or it increase verbosity. But they may be completely ignorant of how the operating system's tcp stack, memory management or other mechanisms work. Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid?
If you have a developer who insists every issue is the operating system, sometimes the SA has to know how to dig in and run stack traces, probe tools (systrace, dtrace, whatever), jmx queries, etc until they can pinpoint the offending code.
As another example if you have an application that isn't draining queues quickly enough and therefore sending back tcp zero window frames upstream, what's the solution? A hypothetical lazy developer will say "it's the OS not queuing enough data, increase the OS buffers." A hypothetical lazy SA may say "it's the app not consuming packets quickly enough, rewrite the app."
In reality if we've all been paged to a priority one bridge the solution will probably be the combination of the two - tactical fix of increasing buffers to create some time for development to understand why the code isn't doing what it should and fix it.
It's funny that you bring up CMMI in the discussion. CMM(I) is nearly the antithesis of Deming's approaches toward quality management. What's really interesting is that Deming's approaches were adopted by Toyota decades ago to historically great effect (sadly with few other large examples of named successes in the business world) while Taylorist approaches (including CMMI) a handful of other Japanese daibatsu and especially US companies back to the 19th century. These companies have had vastly different growth trajectories over time, but when it comes to quality most consumers in surveys will associate Toyota with it over Hitachi, Mitsubishi, and Fujitsu (I believe all of these companies are full-blown CMMI 2.0+ adherents and champions). Similarly in the US, what has Six Sigma really done for companies that have adopted it? GE is hardly known for anything in the public eye resembling technical chops, for example, and most studies show that companies that adopt Six Sigma more than 70%+ of the time lag the S&P 500 upon adoption with no long-term recovery afterward either (perhaps Six Sigma adoption is not a cause but a simple correlation with poor performance similar to private equity oftentimes getting a bad rap in the public eye).
When even the US military - one of the world's foremost investors in management and leadership research - has largely abandoned command and control (the military equivalent of Taylorism) we really need to ask whether structures that enforce a management/worker caste vs. one that empowers those closest to a problem are effective beyond any meaningful scale.
No doubt valid points about CMMI; I was exposed to it as part of a program to improve quality and in that specific instance it was constructive. However the program office oversight was shut down as all portions of the business were certified as "level 2" and without that level of structure and control most of the process died within a year.
Even so my original point remains - in some kinds of highly regulated shops there's enough external pressure for controls and separation of duties that the developer simply cannot have access to production. I'm not defending either practice (CMMI or seperation of duties), I'm just saying in some places it's reality, regardless of perceived drawbacks or overhead.
> Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid?
I've not seen a professional developer be confused about Linux using otherwise idle memory to buffer, no. I have seen that with sysadmins who mostly look after Windows boxes and were somewhat unfairly dropped in at the deep end.
I've seen JVM heap OOM errors be caused by both object management issues and applications that would legitimately benefit from larger heap sizes. Many, many times.
I think I'd fall off my seat if I saw a sysadmin use JMX to find an issue. I did see a security guy (so not really a sysadmin, but he was doing a related job) use strace once. He was remarkable enough to have his own Wikipedia page.
> Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid?
Is there somewhere to read about this, I this might have come up with one of our projects. (At the time, from googling, I suggested they try mark and sweep - I didn't really have any idea but was of the opinion they had lots of small objects.) I don't have much experience in Java but was trying to be helpful!
Maybes: Packaging latency in archive formats (compress before upload, decompress after). Network latency on the upload/download. Block IO performance on the server. Virtualization overhead. Memory or processor constraints. Assumption of equivalence is spurious (eg. server is doing multi-architecture builds and full suites of tests including eg. regression tests). Yep, something like that.
Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.
And I've never thrown code over a wall without having tested it in a representative environment.
The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone.
When a sysadmin sees an issue in "their" environment that they understand there's a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it's the most trivial issue the developer has had to resolve that day.
Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them.
Sorry. There must be great sysadmins out there struggling with terrible developers, I'm sure of it. I just haven't seen it.