The Lounge - CodeProject

I hardly trust myself with my stuff.

Although fire/flood can spoil that.

Even theft. Some guys (plural) stole cases a talcum powder off a truck so who knows they might run off with something critical which would not otherwise seem to have value.

IMHO, it's important to also back up the top three or five things in the mind, so that whatever's lost can be recovered with just self effort, even if it takes time. Am just talking of code backups here.

modified 28-Nov-23 10:31am.

If anyone actually expects ANY cloud service to respect their stuff then they deserve the rude awakening.

The Cloud is nothing but another tool/option. Treat it as the be-all-end-all thing the Cloud Sales people market it as then you're just waiting for a rude awakening. Treat for what it really is and you'll avoid these headaches.

Keep in mind of course that one should consider carefully what 'backup' actually means.

Specifically even though it is seems to be working, is it being verified on a regular basis? So can one actually get to that old data?

jschell wrote:
Specifically even though it is seems to be working, is it being verified on a regular basis? So can one actually get to that old data?

Even though that's common knowledge, I have little reason to believe even someone as big as Google actually does verify its backups.

jschell wrote:
Specifically even though it is seems to be working, is it being verified on a regular basis? So can one actually get to that old data?

A former employer had a rigorous backup schedule for the office file server. Incremental backup Monday through Thursday, Friday was a complete backup that was saved for 12 weeks (tape cartridges were recycled every 12 weeks) and the last full backup of the month was saved for 12 months. We had a collection of 30-40 tapes that were all well-labeled and cycled through the process consistently.

This system had been in place for 5 years and ran flawlessly.

Then my team had to recover a file from a backup. We spent 8 hours trying to pull that file. Then any file. Then from other backups.

We had a box of tapes and not one of 'em was readable. Completely worthless. We reported this to management.

They replaced the system a year later ...

It gets better ... this was in the time when 386 was the main PC architecture. The backup system was a board that only ran in a 286, so the office used an ancient (well, ancient in computer years) PC that we could not update because the software that came with the board didn't run on newer OS. The office was on Win 3.11, can't remember what the 286 box required.

I'm still reading that those who need to perform massive backups still prefer tapes. I've always been a skeptic.

Isn't everything about a tape backup solution proprietary?

If you have a specific type of tape, you have to have the matching drive. The reverse is also true, if you have a specific drive, you can't just use any tape in it. I suppose there might be compatible makes/models (of both drives and tapes), but things are still not as interchangeable as "hard drive A vs hard drive B".

And then the software tends to be proprietary, and not every OS can just read any random file from a given tape. If you've done a backup with software XYZ, you have to do software XYZ to do the restore. Imagine finding a bug in the restore portion of the software, and the manufacturer has gone out of business years prior.

That would worry me, unless I had an unlimited budget and could purchase all the redundancy I'd need to be comfortable. How many tape drives is that? I have no idea. Imagine having stocked up with 5 spare tapes drives...with a SCSI interface.

Obviously big business has different needs than I do. I'm perfectly happy doing "xcopy backups" to multiple hard disks. No proprietary software, you can access any file in any folder instantaneously, no matter what OS you mount the drive with (within reason), hardware interfaces will remain standard for years if not decades before being deprecated, you can buy them in various capacities from different manufacturers, and they're relatively cheap.

I tend to buy them in sets of 3 - one live, a disconnected backup, and an off-site backup. When I outgrow a set, it becomes more or less a permanent archive of "things as they existed at that point in time".

When SATA became the standard, I had plenty of time to migrate the data from IDE drives to SATA drives. Whenever SATA gets replaced, I'll just do the same.

The non-starter is when you have massive amounts of data. I try to keep my entire backup set on a single drive, and I currently have a trio of 16TB drives. If my archive suddenly grew to double that, I wouldn't have a "single-drive solution", as 32TB drives are still a long way off. I'd hate to have to maintain a RAID, and multiple backups of it...

You are pointing out something that is very essential. As I said in a different post, I guess there are millions of old backups that are useless, for the reasons you mention. In addition comes what you are not mentioning: Having the right software (that includes the OS) to run the application to interpret your files.

But I'd also like to mention that there were standard even in the old days. Open reel 1/2" tape formats followed international standards from the mid 1960s. The standards passed then was essentially to accept the IBM 'proprietary' format as the standard - which a lot of other manufacturers already had done. You certainly could move a tape from one machine to a different one.

QIC (Quarter Inch Cassette) tape was standardized in the early 1980s. The good thing is that there are so many QIC standards to choose from Smile | :)

, but the standards were vendor independent.

We are all familiar with memory sticks: The USB standard defines a mass storage virtual drive. SCSI did, too, in 1982. You could plug a disk or a tape station into any SCSI socket; it would identify itself as a disk drive honoring the virtual disk specification, a tape station honoring the virtual tape drive specification, and so on. There were virtual scanners and printers, too, in the specification.

So it wasn't all proprietary. If you have an old QIC tape, you must find a physical QIC reader; it can be of any make, as it handles the one format on your tape, which is by an international, vendor-independent standard. There are quite a few of them, all international standards. Make sure that your QIC reader is a SCSI one (most of them were). Then you can plug it into any PC with a SCSI interface, which can be found almost anywhere, right? Smile | :)

It can be of any make. Windows, *nix and other OS-es see the same files on the tape.

But cross your fingers for that SCSI interface: I was actively in favor of SCSI for a few years: One great standard for everything! It was daisy chained, not a tree, but the chain could have all sorts of devices in it, much like USB. Except that when I had to buy an adapter cable for the eight plug 'standard' for SCSI, and the salesman said I was lucky: There are fourteen different plugs used with SCSI! (this is 25+ years ago; I guess there are more now), then I called a halt. No more SCSI for me.

USB was sliding down that same slope: I've got A, A 3.0, B, B 3.0, Mini B, Micro B, Micro B 3.0. I have equipment requiring all of these, then there are some I never used. When C arrived, I said: OK, this is your last chance. If you come with anything new to replace C, you won't. Not in my house. Then I am through with USB, like I was through with SCSI.

I wish there was a (non-proprietary) standard. There are loads of (non-proprietary) standards - that's the problem.

Of course we cannot stop progress. We must accept new standards. If you apply for a patent, your new invention must have a minimum 'invention height'. It must add something 'significant' (as far as I understand, this is not as strongly enforced in the US of A as it is in Europe). We should follow the same principle in standards. B micro did not add anything 'significant' to B mini, and should have been rejected. Smartphone SIM card come in (at least) 3 different sizes; these are certainly not distinguished by significant 'invention height'. In software: We should accept a new programming languages only if it adds significantly to the ones we have. Similar with Linux file systems (although there is a natural limit of 42 alternatives that probably won't be exceeded).

I do not see 'proprietary' as a big problem. A much bigger problem is that various vendors who could have used an existing standard rather makes a new one - and we accept it, because it it non-proprietary / 'open'. Even if there is no need for it.

dandy72 wrote:
Isn't everything about a tape backup solution proprietary?

Up until about 20 years ago tape was the only solution. So that really wasn't a consideration.

Not to mention that automation systems even with tape were very expensive. So that meant a IT guy needed to swap tapes in and out.

dandy72 wrote:
The non-starter is when you have massive amounts of data

There are other potential problems. For example a database with encrypted columns. Where is the key?

Or an encrypted file. Where is the key or passphrase for that.

Those are not necessarily relevant just to disaster recovery either. If you have 7 years of backups, which is often considered a norm for financial data, and you get a subpoena you will need for legal reasons to show that you tried to recover that data.

At one company I considered disaster recovery just not possible. The DBA stated it would take more than a week to restore the database from the back up. But in addition there was no back up for the proprietary machine that ran the database. I figured best case scenario was that it would take 6 weeks to recover if the hardware failed. They originally budgeted an onsite back up machine but that was nixed due to the cost. Nixed by those above operations/development.

jschell wrote:
Up until about 20 years ago tape was the only solution.

I would raise that to 30 years, if you talk about the only solution. 21 years ago, in 2002, my employer at the time - the National Library of Norway - threw out their tape robot, replacing it with a big wall of RAIDed hard disks. Tape was considered an outdated technology; certainly not 'the only solution'.

So that meant a IT guy needed to swap tapes in and out.

This tape robot, thrown out in 2002, was fully automated. No manual handling of tapes. I am not sure how old the robot was; it had been there a few years when I entered the job. Their policy was to migrate all archived data to new media every 5 to 10 years, and the move to RAID disks were considered overdue. So I guess the tape robot was, at the very latest, installed around 1995.

For example a database with encrypted columns. Where is the key?

Certainly an essential question. But it is the same regardless of which backup technology you choose.

With one slight modification: When I am in physical control of the backup media, I consider encryption for the purpose of the backup itself to be less important. If I were to send the files to some other backup service, there is a lot more that I would encrypt. But I take your scenario to mean a database encryption that is done in any case, not just for backup purposes.

trønderen wrote:
Tape was considered an outdated technology; certainly not 'the only solution'

In terms of cost and process it was about 20 years ago that a transition became possible.

For the example you gave they were already spending a lot on the existing system and the new system would have also have been expensive. Smaller organizations could not have afforded either solution. And there are many more smaller organizations than larger. So in effect manual tape was the only solution. Even more so since at that point even small companies started increasing the amount of data that was stored digitally (versus paper.)

trønderen wrote:
But it is the same regardless of which backup technology you choose.

Yes but my point was more about the reliance on older back ups. So even if the technology was till accessible the information to use it might no longer exist.

jschell wrote:
Keep in mind of course that one should consider carefully what 'backup' actually means.

I've got friends who state in dead earnest that 'I have a backup copy of my photos in the the cloud, so I have cleaned them out of my hard disk'.

I have tried to give them a lesson in what 'backup' means. Not all of them are willing to understand.

I have had to explain to people that a source control system is not a back up.

I think all my programming friends - including those who are programmers by profession - need to have that explained!

We ran a computer and network service company in a resort town and had various customers with a variety of backup software from Windows Server backup to Acronis and while we showed them how to daily check the backup logs they all opted for up to do it remotely for 25.00 a week. It's incredible but I have always contended after babysitting "normals" who have computers is that they shouldn't have em. Primarily because if they don't make image backups they could loose everything when the drive goes kaput. Todays SSD give no warning but just leave town with your deftly arranged bytes that look like what you care about.

Now not only is Microsoft taking your libraries and putting it on there "One Drive" computers re-pointing paths to that so it;s "transparent to the user", Now they are bitlocking it too. Try to have a conversation with your Dad about how MS encrypted your data, they have it and without the 48 bit key to unlock it...... They look at you like you are from another planet.

I'm done.

To this day, I maintain that I had stuff on One Drive that simply disappeared ... while the folders remained. So, no, I don't rely on the cloud for "backups".

"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I

The thing is, there isn't just one cloud. If you're not precious about them having your files, backing up stuff to TWO (or more) cloud services would seem to be a robust option. Even if MS lose your data today, it's pretty unlikely Google will too. Voila! Redundant off-site backups.

Telegraph marker posts ... nothing to do with IT
Phasmid email discussion group ... also nothing to do with IT
Beekeeping and honey site ... still nothing to do with IT

I never trusted online backups. I do not trust them to keep my data safe. Nor to keep them private. Nor to allow me to store arbitrary files. Nor to be able to present my photo album to my grandchildren 50 years from now. Nor to not show them my very personal notes that were meant for noone but myself.

I could use an online backup to save an extra backup, in case my primary backup is lost. A lot of files in that extra backup would be encrypted before sending them down in the crypt. I am not currently doing that.

Does the online bakcup have an offsite bakcup? (Thanks to fgs1963 for reminding us about offsite copies!) This story seems to suggest that the answer is 'No'.

There are a couple other points that are often overlooked: I have some old files backed up on both DC100 and DC300 quarter inch tape cassette. (Actually, I even have some files on 1/2" open reel 7-track tape, but I made that tape mostly to learn how to handle the tape station, not really for bakup purposes). I have files on two different Travan tape cassettes. I have at least one hundred 8" floppies (they were really flopping!). The first digital camera I used saved the photos on 2" floppies - I still got the floppies, but the camera belonged to my workplace, 30+ years ago. I never had my own Travan station; it belonged to my workplace, too. Obviously, I didn't have a 7-track tape station at home. I could have had a DC300 (the format changed name to QIC); they weren't that expensive, but I rather brought my private files to work and saved them to tape there.

I guess that if my life depended on it, it would be possible to have the files on QIC tape recovered. It probably would take too much time to save my life. It probably would be terribly expensive. But there were a whole crowd of variants, in track layout, packing density etc., so don't expect just any old QIC station to be able to read my first-generation DC300 tapes (before the QIC standard arrived). The open reel and Travan tapes, and the 2" floppy, are nice computer museum artifacts. Even if my life depended on it, I guess I would have crossed the river before anyone could get hold of a reader for them.

I actually have both 5.25" and 3.5" floppy units, but I am about to ditch the PC that can handle them; I haven't booted it for years. Most likely, some of my friends still have 3.5" units, Maybe even 5.25", I wouldn't be sure of that.

Then comes the second major problem:

I have migrated 'the most important' files from one medium to another. Over 40 years, I have created documents in more than a dozen different formats. Note that some filename extensions (such as .doc) hide several generations of formats which may be quite different, but new software versions handle all the old formats too. For photos, the number of formats exceed two dozen. Sound: Somewhere between one and two dozen. Video: A handful of different raw formats. At least four 'final' formats. I've got analog tapes with no noise reduction, with Dolby B, Dolby C and open-reel tapes with dBX noise reduction.

Some of my amateur photographer friends insist on saving all their photos in '.raw' format, seriously believing that there is One Unique and Never Changing Raw Format. That 'raw' is a totally unambiguous concept. It isn't.

I am quite sure that in basements, attics and safe deposit boxes, there are millions of diskettes, tapes and other storage media for which the owner no longer has a reader. If he gets hold of one, there is a great risk that he won't have the software to read the media (there are other file systems than FAT, NTFS and the 42 different *nix file systems!), or if he can copy the files to his current PC, that he has no software to display or play the files.

So, a great deal of the backups made are useless. Cannot be retrieved, or retrieval will be tremendously expensive. So maybe it is a good thing that 'Social' Media teaches us to treat anything that is more than a week old as outdated and irrelevant. No modern grandpa or grandma digs up an photo album from their childhood, to show and tell the grandchildren how it was back then. The old ones never even suggest it, and the kids would find it boring. So it doesn't really matter if the files are lost.

Quote:
No modern grandpa or grandma digs up an photo album from their childhood, to show and tell the grandchildren how it was back then.

We do. I even digitized old super 8 films, put them on a web server (Raspberry Pi) and put it on line for them to watch. They said they did. The wife is making copies of photo's and newspaper clipping for my daughter to take to her 30th HS reunion.
Alas, born of the 30's, we are string savers. We each vow to go first to avoid having to go through the other's "stuff".

>64
Some days the dragon wins. Suck it up.

trønderen wrote:
and the kids would find it boring. So it doesn't really matter if the files are lost.

Yep.

The fact that one person finds is significant and valuable does not mean that others will.

The only difference with digital is that it will give someone the chance to toss it without agonizing over whether they should keep it or not.

trønderen wrote:
Some of my amateur photographer friends

I had an office mate who had 1,800 photos of his 18 month old child. All labeled. All digital fortunately which made getting the count easier of course. (Even worse I can't remember the exact number but it might have been 18,000.)

Second child was on the way.

There is a well known phenomenon where parents take way more photos of the first child than the children that come after that.

When digital cameras arrived, I was not first in line to get one. Two of my co-workers were, long before me. We had a social mountain cabin weekend at work, and Monday morning, one of those digital guys put his 700 photos from the weekend on the company file server. The other digital one followed up, publishing his 500 photos on the company server.

When I had my 36 slides developed a couple of days later, I didn't care to try compete with the 1200 photos already on the server (actually more: Several other co-workers had added their 100, 150 or 200 photos.)

On the other hand: The silver photos I made are still among those I use to show what kind of pictures I am proud to make.

trønderen wrote:
one of those digital guys put his 700 photos from the weekend on the company file server. The other digital one followed up, publishing his 500 photos on the company server.

lol...just one weekend?

Did they even do anything except take pictures?

Having cleaned up after 3 different ransomeware attacks (different clients), I probably have different thoughts about backup. Worst loss: 3 days of email. All backups were air-gapped.
There is Backup and there is D.R. Different requirements, IMO.

>64
Some days the dragon wins. Suck it up.

This one's going to run and run, but what I do is:

daily backups to a second, local hard drive
daily backups to OneDrive (I use that because storage there is cheap)

I feel on pretty safe ground here, don't reckon I'm going to lose anything. I used to also back up occasionally to an external drive, but I don't do that anymore.

Paul Sanders.
If I had more time, I would have written a shorter letter - Blaise Pascal.
Some of my best work is in the undo buffer.

Welcome to the Lounge