How do web applications explode out to 3 Million lines of code? Yelp, to me, looks like a typical CRUD app and I would have been surprised if it were more than 100,000 lines of code. The software I develop is pretty large and typically doesn't surpass 40,000 sloc written in-house (i.e. excluding third party libs).
Does anyone here maintain such large codebases? Are they truly that big or are people just counting third party code and generated stuff?
I think you underestimate the complexity of things that happen under the hood or simply out of sight: back-office apps, integration systems for data import/export, backups, alerting.
I'm not so familiar with Yelp, but I'd bet they're interfacing with a crap load of additional stuff, including probably in-house tools to follow leads, facilitate reviews, handle their ad programs, etc...
I've commonly seen codebases explode the 1M LoC mark. Not necessarily for a single component, but if you have multiple systems interfacing with each other it's really quite common for business applications. That's obviously excluding libs.
Frameworks are a bit responsible for this in my opinion (note: not saying frameworks are bad, but it's a side-effect), as you'd often either have some additional configuration, boilerplate code, or generated code.
Also, it your application is long-lived (think decades), it's even less surprising: new engineers come and go, and it gets harder and harder to touch the things that were maintained by the previous key-holders... So you had a new stone here and there, polish a turd here and there, but you don't really untangle the mess that sits right at the middle, because it's just way too dangerous (or so you think). And it goes on and on. And it's aggravated by the fact that, as the project grows larger, the barrier for entry for new developers get higher: it takes longer and longer to understand the system/platform in depth, and many never even try to get there.
Processes and security concerns affect this too: you're often only allowed to fix something which has a ticket assigned to it, originating from a business user. Touching anything else is a big no-no, as it would mean QA has to re-test all the things that could be impacted. Of course we can argue whether that's actually the case and how proper testing would mitigate this, but you see my point...
I maintain a line-of-business webapp that could be mistaken for a typical CRUD app, but actually has a lot of business logic enforced in code.
That's stuff you don't have to hardcode -- you can pull it out into a 'rules engine' (at the expense of an additional runtime dependency) or push it further down into, say, database stored procedures (I can hear some of you shudder). But for us, the rules rarely change, or change at a pace that's acceptable to keep up with.
Also, there are a lot of views and specialized interfaces tailored for particular workflows. In several cases, the data underneath of them is the same, but there are different UIs -- that adds LOCs considerably.
I wouldn't draw the distinction at 'hardcode' or some alternative. Rather the question is which language expresses the business rules well enough (or even best).
If your application language also does a decent job of expressing the business logic, that's great.
The difference is, I think, that you might want to be able to ad-hoc configure the rules without changing code. Going beyond that, you might want to allow people who are not developers to add/remove/change rules around business logic. Building all of this up can be quite an effort depending on how complicated the rules get.
I don't know exactly what Yelp does, but assuming that they have listings for restaurants that aren't their customers, one cause could be that they take in data from lots of sources. Ideally, that is all in the same format, with an enforced data scheme.
However, if you require that, you'll notice that very little data manages to make it through your entry port. Few suppliers will want to bend their system for you to comply with your rules, and those who do will find inconsistencies in your specs or just ignore them (different encoding, not sending state abbreviations separately from city names, forget to encode HTML entities, encode them twice or thrice (my personal record find here is four levels of HTML encoding in data stored in an excel file), etc)
So, you end up with quick and dirty hacks that convert the suppliers format (or, rather, what you _think_ it is, as the supplier won't be able to tell what their format is, either) to yours, fix some egregious errors, etc.
Hundreds of data suppliers, code with tables mapping their codes to yours, or that map known errors in their input to corrections, 'smart' programmers who notice that they can replace that 10,000 entry table with corrections with 20 lines of code and a 1,000 entry table, except that, a few months on, that 20 lines have ballooned into 1,000 that nobody understands, so that this code can't be used for handling the data from a new supplier anymore, and the line count starts increasing rapidly.
On top of that, once you operate world-wide, you'll learn the joy of differences in addresses. Does a country have states? Zip codes? If so, where does one specify them in an address? If you want to localize that in your app (in yelp's case, people may want to show an address to a taxi driver. For that, it would help if the address followed local conventions) line count skyrockets.
> On top of that, once you operate world-wide, you'll learn the joy of differences in addresses. Does a country have states? Zip codes? If so, where does one specify them in an address? If you want to localize that in your app (in yelp's case, people may want to show an address to a taxi driver. For that, it would help if the address followed local conventions) line count skyrockets.
This is not the first time this issue has been faced. Why would they reinvent the wheel instead of just using libraries and conventions?
This is something that grows slowly on you, and it is a very hairy problem. By the time you realize the scope of your problem, finding a library that can be shoe-horned into your code and doesn't have huge regressions on your data is hard.
Also, assuming a robust generic solution exists, it will almost certainly be slower than a customized one. It is very tempting to think "but our data is relatively clean, we don't need the full feature set of that library that makes imports take an hour longer to run".
When you don't spend the time to refactor and groom the codebase, actively seek to reduce the complexity, it grows. And it grows exponentially. Once you reach the point where you're afraid to change something because it might break something unrelated, you just add new code all the time.
I've worked at a couple of places that went this way (joined after they already had millions of lines).
It was code duplication. One example I saw was a many thousand line css file included 4 times in a row. They were all different generations and some were customer specific so it was an impossible task to try to consolidate.
That was a common theme. Enterprise stuff where one customer needs slightly different behaviour so code is duplicated (maybe thousands of lines).
The first time you take that road, you've doomed yourself. With a good / small team they can manage it- but imagine when you've got 20 devs sitting there of varying experience. The precedent is set and the solution for anything mildly complex will forever be, make a copy.
Oh man, I would love to work at a place like that. If only there was a "Sandi Metz" specialization in software development. The worst part might simply be managing the egos of people whose work is refactored.
You can find these places everywhere. Look for jobs where the software department isn't the core business and you'll find small groups of devs bogged down trying to implement another customization among code that looks like
switch(clientId){
case 42: //todo: Make sure database agrees
...
}
before each calling what are essentially copies of otherwise identical thousand line functions.
Because the "risk" of actually changing code for any client where things already "just work" is a testing risk, and testing is nearly always the bottleneck at these places in my experience, if they've even managed to hire a dedicated tester yet.
Of course if you apply the Joel Test you're likely avoiding these places. But they can be rewarding places to work.
Is the 3M across all applications they developed? That may be much more than just the customer facing part. Plus like others have said, it's also years of development; I've heard a lot of stories that a lot of the high-profile companies and applications (paypal, ebay, linkedin, skype, etc) have huge and rather bad codebases.
I work on a product in payments, specifically a client-side checkout application. It's essentially responsible for collecting data from the user (billing and shipping addresses, affordability information, etc.), presenting the various payment methods and displaying up sell stuff after completing the purchase. Just the client side application is about 250000 lines of code, not including third party stuff.
It feels like applications grow with the team that develops it. At our peak, we were around 10 people working on it, and it's just so hard to do meaningful refactoring when large parts of the codebase are being modified by 10 people in parallel. Not only the technical challenge, but also getting everyone to agree on and embrace patterns and structure.
They probably only add code and never refactor or delete it (and by the way I don't think it is always bad idea if the code is organized). So a large team can write a lot in several years.
This code might also include HTML templates and CSS or JS files that can easily grow large.
I'm looking at one right now with 2m+ lines. It's more surprising that they don't have closer to 10m to 20m lines by now (3m feels smallish for their scale).
Also it's easy for business apps that have been around for 10+ years and continually worked on during that time to get to 1m-2m+ lines of code... and jump from 2m to 3m is easy too.
We have an in-house configuration system that generates configurations for about ~14k heterogeneously configured applications (1-3x 3-20 line config file per app) and that system is has about 451k lines in it.
Of course not all lines are code. About 140k lines are flat file database of configurations, and the other ~300k lines is code/templates.
300k lines of code to manage 14k applications. lol.
------------------
Actually did a more thorough analysis cleaning out comments and what not. The database is more manageable ~86k lines and template generation code is around ~47k lines. There's a crap ton of whitespace and comments.
Yelp has 4154 employees and probably 50% of them write code. Imagine you have 100'000 customers who all want 2 custom features. Large organisations naturally develop large codebases. Yelp probably has 50M LOC and more. Probably most startups could do with 10x smaller codebases: http://www.paulgraham.com/avg.html
I have been watching an e-commerce open source product grow over the last four years. The product is getting really big. Part is the amount of features but the other part is that there is a lot of code duplication. Developers are focused on adding features. No one seems to care about removing code. I suppose they do a lot of copy and paste. It just grows and grows.
Previous gig I worked at had a decade old codebase for just the web app that was a bit over 2.5 million according to one of the chief engineers. That wasn't even the complete product just a small piece.
Does anyone here maintain such large codebases? Are they truly that big or are people just counting third party code and generated stuff?