|
Glued together from a web of dependencies, development "organized" via agile methodology.
How long before we just fire software packages out of a t-shirt cannon?
"Hold my beer, I've got a version update to deploy"
I am really uncomfortable with the state of software development these days. Dependency hell didn't used to be a distributed problem.
I suppose I'm just getting old and this is the new normal. Eventually we'll *need* AI just to keep our import hierarchies straight. 
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Forget the t-shirt cannon, if you go to QA you will see that the current development methodology is the "software party popper": grab a dozen small chunks from the internet, chuck them all in a file, and watch your software explode... 
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
People complain about dependency hell, but they forget the alternative being that you would have to write everything yourself from scratch. I'm sure for some people this is fine ("HTTP server? I'll do it myself!") but for a large majority of people this isn't going to cut it.
I use Java and Maven to manage my dependencies. If I choose to upgrade my dependencies because it's overdue, does it cause me headaches? Absolutely! I might have to spend an hour or two carefully adjusting versions until everyone is happy. But the trade-off is that I haven't had to build entire frameworks from scratch, and would I even want to do that if I could? Absolutely not.
|
|
|
|
|
The problem is not the presence of dependencies, but the number of dependencies. Instead of using a single framework and then writing the things missing in this framework, many programmers will import a package for each minor piece of code.
Leaving aside the security issues (without reading the code, how do you know that the package doesn't contain malicious code in addition to the useful stuff?), the maintenance overhead of so many packages tends to be prohibitive. I would prefer to use fewer packages, and not have the headache of working out whichg version of X is compatible with Y, etc.
I won't comment on the practice of some developers of making breaking changes to their code - either by renaming functions or by changing the functionality of existing functions. Any such comments would not be KSS.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
I mean, hey, people obviously weren't having enough trouble managing local dependencies, so now we've flung them across the Internet.
So when your guy making a Widget control goes through a messy divorce, maybe one of the things he does is pull his project.
Then what?
Like I said, dependency hell didn't used to be a distributed problem.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
What are the options then? You either:
- Build your own libraries from scratch and maintain them yourself, potentially taking weeks or months of work
- Manage your own dependencies with version control by manually downloading whatever you need and having them copied into your projects
- Use dependency managers like NuGet, Maven, Gradle, npm etc
You make it sounds like there is no winning solution. I'd much rather make use of a dependency repository and know I can almost click + collect a solution to whatever problem I have, than spend a long time Googling for an answer and then trial and error'ing until I find a working version.
And the issue of people pulling libraries is a non-starter if you're using an appropriate dependency proxy. In Java we have Nexus, which acts as an intermediary which downloads and caches dependencies, so even if they're pulled from the internet (or someone maliciously overrides an existing version) we have constant access to what we need.
I'd be more than happy to hear alternatives to dependency/package managers!
|
|
|
|
|
How about not using dependencies for simple code?
Not using dependencies just because you can.
That's a start.
Chris Copeland wrote: And the issue of people pulling libraries is a non-starter if you're using an appropriate dependency proxy. In Java we have Nexus, which acts as an intermediary which downloads and caches dependencies,
This is actually hilarious to me. It reads like "It's not an issue. You see, it became such an issue that Oracle caches dependencies to prevent it"
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Of course you shouldn't be using dependencies for simple code, that should be a given. But I build a lot of microservice APIs, and each one has different requirements. Of course I'm going to use dependencies for those, including HTTP servers, security etc.
You could argue "well if you're using HTTP servers so much, why not just make your own and re-use it for all your projects?" and you're right, I could have done. But why re-invent the wheel by building a brand new library to do exactly what someone else has done for me.
The point I'm arguing is simply that before we had reliable dependency package managers, the industry was constantly rolling their own versions of things (even downloading other libraries and then making changes/amendments to meet their own requirements). If you hopped between jobs, you'd be learning an all new framework from the ground up, and each one had their own pros and cons. Nowadays, there's industry standards in adopted tools, so finding a new job is just matching up what skills you have. It sounds to me like your argument is focused on solo development, or for hobbyism.
I've worked for several companies as a developer, ranging from small (2-3 team) to medium/large (30+ team, but department of 300+), and can tell you that in each role we used Maven and used a lot of dependencies. Why? Because we were building software that demanded tools that would have made no sense building from scratch when the solutions already existed out there. Try promoting "dependencies are baaaad" when you have to build an inbound rest API that provides credit card processing in real-time with transactional database queries, transaction ingress validation and an external API call to validate the card transaction claims, all while keeping processing speeds to a minimum.
This is actually hilarious to me. It reads like "It's not an issue. You see, it became such an issue that Oracle caches dependencies to prevent it"
You're clearly commenting on something you know little about. Nexus is a product created by Apache for anyone (but mostly commercial) to use as a proxy-cache for dependencies, as well as a private distribution channel for internally built and deployed libraries and packages. The fact that is provides security in caching dependencies and preventing malicious overwrites is just a handy part of the tool, and I'd wager most companies working with Maven are using this tool or similar. You can find similar tools for any package manager.
|
|
|
|
|
Chris Copeland wrote: Of course you shouldn't be using dependencies for simple code, that should be a given.
If it's a given then why do so many people do it, in so many projects? It seems the norm rather than the exception.
Chris Copeland wrote: You could argue "well if you're using HTTP servers so much, why not just make your own and re-use it for all your projects?"
Why would I? This isn't about NIH syndrome. If it was, I would have mentioned it.
Chris Copeland wrote: The point I'm arguing is simply that before we had reliable dependency package managers, the industry was constantly rolling their own versions of things (even downloading other libraries and then making changes/amendments to meet their own requirements).
Oh you're making a point about dependency package managers? Because I'm not. My point is about people.
Chris Copeland wrote: You're clearly commenting on something you know little about.
[Then goes on to explain that it's exactly what I said it was]
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
If it's a given then why do so many people do it, in so many projects? It seems the norm rather than the exception.
I'm guessing for convenience? If a library exists out there that accomplishes exactly one of the things that you need your solution to do, why would you not use it? If I wanted to build a quick tool that processed some CSV file and dumped the data in a database, I'd probably leverage the two dependencies I am familiar with which will expedite this process, cutting my development time down from hours to, likely, less than an hour. And you're probably thinking "well this is my point, people just jump straight to using dependencies and that's why we're in this mess", and absolutely, you're right. But again, people also don't want to be re-inventing the wheel and wasting hours or days building a new library that does exactly what another library does.
Why would I? This isn't about NIH syndrome. If it was, I would have mentioned it.
Then what is your argument here? I get you're saying that reliance on dependencies can make projects messy, and that maybe we're overly dependent on them, but what's the alternative? Does everyone go back to writing their own libraries and storing them in version control? It's like an anarchist saying "the government is terrible and we should abolish them!" Okay, if I agree that the government is terrible, what is the alternative?
Oh you're making a point about dependency package managers? Because I'm not. My point is about people.
Well package managers are where people get their dependencies, unless they're downloading the sources and building them themselves. People have unified repositories for downloading the libraries they need, rather than ye olde days when you'd manually download DLLs, or checkout code and build from source, and store in version control. People choose to use these package managers because they can Google "c# asp.net validation nuget" and be presented with packages which provide validators for ASP.NET modal objects.
[Then goes on to explain that it's exactly what I said it was]
Except it isn't exactly what you said. Oracle never developed anything, it's an Apache tool which is used by businesses and freelancers for storing and caching dependencies on a node within an infrastructure to reduce network-load in both build pipelines and on internal networks. The fact that it provides additional security by preventing malicious version overwrites, and retains dependency versions even when removed from the source distribution channels, is an added bonus. Is it an added layer of complexity? Sure, but it beats the hell out of keeping your dependencies in version control, or sitting in a folder somewhere on your PC for when you want to use it.
I'll throw you a bone. Do I agree that dependencies are an absolute mess? Of course I do, my projects have hundreds of dependencies because the base packages I use also have dependencies, which have dependencies. The fact of the matter is though, everything I use is industry standard in my field, if I look up job descriptions for my role almost all of the listings expect experience in the libraries I'm using. And I'm not managing all of these hundreds of dependencies, I manage about 25, the tools I use take care of the rest. If I don't use these libraries, I have to build an insurmountable of code to accomplish what I need.
And as I said before, if I choose to upgrade my libraries I do have to contend with conflicts. But I'd rather spend an hour carefully going through and getting my versions right, than spend months building my own frameworks and libraries to accomplish what already exists.
|
|
|
|
|
Chris Copeland wrote: why would you not use it?
Because there's a cost associated in terms of relying on other people's code. You have things like trust, control over the direction of it, bugfixes on your schedule rather than theirs, and a myriad of other reasons.
Using a dependency is a *decision process* and I'm glad you asked this question because it basically answers a question you asked further down, to wit - "what are you arguing?" so I'll address that here.
Chris Copeland wrote: why would you not use it?
If you have to ask that question, then it sounds a whole lot like you don't acknowledge that there is effort, work and compromise in managing external resources, not just internal resources. There are plenty of reasons not to use dependencies depending on what it is, and the situation.
And this attitude of ignoring that is exactly the problem. Dependency hell is real. Now it's a distributed problem which makes it worse.
Chris Copeland wrote: Well package managers are where people get their dependencies, unless they're downloading the sources and building them themselves.
It doesn't matter how people get them. What matters is how and when they use them. See above.
Chris Copeland wrote: . Oracle never developed anything, it's an Apache tool which is used by businesses
Oh Apache, not Oracle. My mistake. Guess that renders my entire argument invalid then.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Because there's a cost associated in terms of relying on other people's code.
Sure there is, but there's also a significant (and real world) cost to relying on building your own code-base and tools to accomplish these things. I joined my current company last year and was tasked with building them a sales CRM system from scratch. No existing IT department, no existing software, no infrastructure. Once we'd solidified the base requirements and it came time to getting started, I chose the frameworks I was going to use, got them setup and within 4 hours had a running HTTP server with full OAuth security and barebones frontend.
Now if I had come into the business and said "okay well it'll be best if we roll our own HTTP server, our own frontend framework, etc. etc." it would have been months before I could even get started on the application itself, and the business would be down many months of my salary with nothing to show for it. And, could I even reliably build an entire framework which has all of the security and features of the library I was using? Probably, but to fully replicate what I needed would have taken much more than months.
Of course I recognise that there's time, effort, money and compromise involved in using dependencies, but that's what companies and development teams do. They weigh options and discuss the pros and cons before they choose what to use. As a tech lead in a previous role I was constantly being asked which libraries we should use for what, that's just part of the role.
It doesn't matter how people get them. What matters is how and when they use them. See above.
Does it matter so much? If ol' Joe Bloggs is working on a personal project and he decides he wants to use an existing library to cut down his development from days into hours, I find it hard to understand what's so terrible about that.
If a company's development team collectively agree to build their new greenfield project using X library with Y and Z dependencies so that they can have it completed within 6 weeks rather than 6 months, is that so horrible?
I mean, ultimately, you can choose not to use any dependencies at all, and completely build your own libraries and keep everything internal. Absolutely nothing is stopping you, and I do see you build your own libraries and share them on CP, which is fantastic! I myself have built a few libraries too and shared them, just for kicks.
But the reality is, the development ecosystem has moved on from just building their own tools and managing the code themselves. I like to think of it as being in a stage where all those incredibly complex problems we used to commonly face now have pre-built solutions, allowing us to get on with actual development, rather than breaking our backs trying to put the base pieces together. If my colleague can ask me to quickly throw him together a landing page for some marketing leads, and I can do that within an hour or two because I have all the tools readily available for me, then I count that as a win. If that project ends up with 100+ dependencies, I'm not managing each of those dependencies, I'm handling maybe 5 or 6. My development tools handle all of the rest.
|
|
|
|
|
Chris Copeland wrote: Sure there is, but there's also a significant (and real world) cost to relying on building your own code-base and tools to accomplish these things.
I never argued there wasn't. In fact the only reason I didn't bring it up is because it's a painfully obvious point that I felt didn't need to be reiterated. Also, water is wet.
Chris Copeland wrote: Of course I recognise that there's time, effort, money and compromise involved in using dependencies, but that's what companies and development teams do. They weigh options and discuss the pros and cons before they choose what to use.
I wish that was true. God, how I wish that was true. If it was true in general, NPM's repository footprint would probably be about 1/3 of what it is. As often as not, the situation is a developer is tasked with adding some functionality to a web application, and they just decide what they're go going to import unilaterally, and how much of that is thought about beforehand depends on the developer alone, often with predictable results.
Chris Copeland wrote: Does it matter so much?
Yes.
If you don't think about what you are writing, you are not writing software, you are free association babbling.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
modified 15-Apr-24 7:22am.
|
|
|
|
|
I never argued there wasn't. In fact the only reason I didn't bring it up is because it's a painfully obvious point that I felt didn't need to be reiterated. Also, water is wet.
If you're going to discuss the costs and implications of using dependency libraries, you do also need to discuss the opposing side of costs and implications of building your own. Just because it's obvious doesn't make it any less a credible argument, companies are not going to be okay with their development teams spending weeks or months building software packages which already exist out of the box.
Just because "there's a risk" when using a third-party library doesn't offset the enormous costs involved with rolling your own. Heck, even if you did roll your own version, there's a strong chance you'd be introducing your own bugs and security issues. Libraries and frameworks which do provide infrastructure (ie. for web, Django, Spring, ASP.NET, Laravel) get regularly tested and probed for CVEs, which isn't something you'd have for a home-grown library.
I wish that was true. God, how I wish that was true.
Every company I've worked with has done this, and companies my friends work for do this. In all the open-source projects I've used, they have code review processes in place which analyse whether changing dependency versions might cause problems. Unit and integration tests are there to test these sorts of things too. I'm not saying everyone does this, it's down to the individual teams and people. It sounds more like you've had personal experience with a couple of bad actors, and are projecting this onto the wider development community.
Just because people are publishing packages out doesn't mean you have to use them. There are a lot of silly solutions to non-existent problems, and a lot of libraries which duplicate things already. But this just leans into the original problem: why are people re-inventing the wheel when there are libraries out there that do the job anyway? It's because some people have this mindset of "I know that library does what I want, but I think I can do it better/I don't like how they did it so I'll make my own", or they feel like they have something to prove and need to build it from scratch.
If you don't think about what you are writing, you are not writing software, you are free association babbling.
And there-in lies what I believe is the underyling tone of this thread. This comes off as saying "If you don't write the software yourself then you're not a real software developer". This isn't even remotely true. In fact, a good developer knows how to best leverage the existing tools available to accomplish what they need. There's absolutely nothing wrong with leaning on existing frameworks and libraries to build what you need to build.
In my previous job I wrote a piece of software designed to issue invoices out to customers, and it processed hundreds of thousands of invoices every day. This included uploading them to cloud storage, updating the customer portal with the invoice details, issuing out emails to the customers, updating internal systems with audit information. I built all of this using existing frameworks (Spring Boot, AWS SDK, Apache Camel, etc.) and it was an incredibly powerful and complex piece of software, but does this mean I did nothing more than free association babble? Am I any less of a software developer because I didn't take the time to build my own cloud infrastructure and asynchronous message processing engines? Drat, I didn't write my own distributed bucket-lock-based SQS, I relied on AWS to provide that for me.
|
|
|
|
|
Nope. It's not that at all.
It's just using dependencies or not should be a thoughtful choice.
For too many people it's not.
I get that you think that's not the case.
What I've seen says your thinking is not in line with the actual applications that are out there. Particularly on the web.
I don't care if you agree. You wanted your argument and you got one.
Meanwhile I said what I came to say.
Pretty much everyone on this thread got it save you.
I'm done.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
honey the codewitch wrote: then it sounds a whole lot like you don't acknowledge that there is effort, work and compromise in managing external resources, not just internal resources. There are plenty of reasons not to use dependencies depending on what it is, and the situation.
I like anti-lock brakes and all wheel drive on my car.
And I absolutely adore the back up camera.
Yes, they are more complex. And definitely more expensive. I can't imagine what it would cost to fix that camera if someone runs into the back up my car but I know it would take very little to destroy it.
But no way am I going to roll my own.
And no way I am I going to look for a Model-T just because it was far simpler.
For that matter I like the lighter I bought just to start my outdoor BBQ. No way I am going to start rubbing two sticks together.
|
|
|
|
|
This isn't about that.
It's about not putting truck nuts on your sedan just because you saw them on amazon.
The choice to use a dependency or not should be a thoughtful one. Not *google google google* "PAY DIRT!" *click!*
It isn't complicated.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
This statement proves my view that most people who call themselves 'developers', 'programmers', 'coders', whatever, shouldn't be developers. They should stop immediately and go become accountants or estate agents.
|
|
|
|
|
Eee-yah. 
|
|
|
|
|
Earlier this week I needed something to quickly decode a string - I don't know how this string I'm handed has been encoded, but I know it's part of some well-supported standard. Anyway, I did find a NuGet package that had the function to break it apart, but it had its own set of dependencies - it brought along 8 new packages (!)...some of which I was already using, but still - had it not been for time, I would've been very, very tempted to roll my own. I held my breath in disgust and they're now part of my solution file...but every time I look at it, I can't help but go " "
|
|
|
|
|
How do you know it's correct? How well does it react to corrupt data? 
I would learn the "well-supported standard" well enough to at least make a rough version of the encoder and decoder, and then compare inputs and outputs between the two. Once you have that in place, if they agree and unless it's seriously slower than the reference implementation, just use yours.
I'd be interested in knowing more about this "well-supported standard" -- last year I had to implement a JSON parser which could detect and react to certain imperfections in some files we were receiving from a third-party utility.
|
|
|
|
|
PIEBALDconsult wrote: How do you know it's correct? How well does it react to corrupt data? 
I don't. And I don't have the resources to investigate.
All I know is that it's a library Microsoft wrote, so I have to trust it (as far as I can trust Microsoft). I'm still not going to name names to protect the innocent.
|
|
|
|
|
dandy72 wrote: a library Microsoft wrote
I'd put money on it then.
dandy72 wrote: not going to name names
 I would have welcomed the exercise.
|
|
|
|
|
dandy72 wrote: Earlier this week I needed something to quickly decode a string
For something similar to which I knew how to write the code, the very best solution I found pulled in hundreds of dependencies. It was part of a much larger application. No way I was comfortable with that so I just rolled my own code.
|
|
|
|
|
Totally agreed. I'll have a look at the actual implementation and, if I'm allowed the time, just might roll my own as well...
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
