1. Welcome to our site! Electro Tech is an online community (with over 170,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.
    Dismiss Notice

Disk compression

Discussion in 'Computers and Networks' started by alec_t, Sep 5, 2012.

  1. alec_t

    alec_t Well-Known Member Most Helpful Member

    Joined:
    Jul 10, 2011
    Messages:
    9,253
    Likes:
    1,218
    Location:
    Cardiff, Wales
    I'm running Windoze XP. Doing a bit of PC maintenance.
    Are there any disadvantages to using disk compression to save disk space (I'm guessing file access could be a tad slower) ?
    Any idea how long the compression process takes?
     
  2. cowboybob

    cowboybob Well-Known Member Most Helpful Member

    Joined:
    Oct 22, 2011
    Messages:
    3,022
    Likes:
    477
    Location:
    James Island, SC
    alec_t,

    Surprised no one's posted (I think my machine is still working:confused:).

    First, determine if your HardDrive is set up for FAT or NTFS file handling. For an XP machine you can only compress the data if the file system is NTFS. Unless you bought it NTFS, or converted it yourself, it's most likely FAT.

    If NTFS, compress away.

    What you gain is storage space. You will, however, experience a slow down in file retrieval due to the OS's need to de-compress the file(s) before you can use them.

    And the whole compression process can be quite lengthy, depending on how much data you have.

    Back when I consulted, I generally recommended against compression of the disk data (as opposed to "zipping" individual file/folder(s) for emailing, posting, etc., and instead suggested additional storage.

    External drives have gotten so cheap (we (Apple) used to sell, in 1984, a 5MB (YES, MB) drive for, retail, $5,000USD!) that's its really easier and more convenient to just slap on an external drive (1TB= <$100), get whatever there is you rarely use on your current drive off onto the new drive, CHKDSK and defrag the old one and be on your way.

    This might answer additional questions: http://www.microsoft.com/resources/...en-us/ff_file_compress_overview.mspx?mfr=true

    CBB
     
    Last edited: Sep 6, 2012
  3. Sceadwian

    Sceadwian Banned

    Joined:
    Oct 27, 2006
    Messages:
    14,047
    Likes:
    141
    Location:
    Rochester, US
    Speaking from experience, compression to gain space is a complete and total waste of time, it almost always has been. The last time I tried it was back in the MSDOS doublespace days, and it was a horrible experience.

    Modern data is typically already compressed to at least some degree so the overhead to accessing it is built in where additional compression which nets you negligible space will just net you more processor overhead. There is no specific method to determine how much extra burden this will place on your system as it depends are too much on the number of cores and types of applications that you typically run.

    As Cowboybob said hard drives prices are so ludicrously low that there's no point. You can buy a 2 terrabyte internal hard drive for 110 dollars. 1 terrabyte for around 85. The processing overhead is very high for the extra space you get (a few percent if you're lucky, and absolutely none for previously optimally compressed data such as video and audio) There can be clashing in compression methods where a compressed compressed data source (such as a video or audio) can actually grow in size slightly due to the required header information.
     
  4. dave

    Dave New Member

    Joined:
    Jan 12, 1997
    Messages:
    -
    Likes:
    0


     
  5. alec_t

    alec_t Well-Known Member Most Helpful Member

    Joined:
    Jul 10, 2011
    Messages:
    9,253
    Likes:
    1,218
    Location:
    Cardiff, Wales

    Thanks for the replies, guys. I was beginning to think I'd posted into a black hole :). You've convinced me not to bother with compression.
     
  6. jpanhalt

    jpanhalt Well-Known Member Most Helpful Member

    Joined:
    Jun 21, 2006
    Messages:
    5,894
    Likes:
    502
    Location:
    Cleveland, OH, USA
    Don't know about your your horrible experience. Please accept my condolences. Can you give some details relevant to 2012?

    There is an advantage to smaller drives in RAID configuration. They rebuild a lot faster. Don't you have intermittent power failures in upstate NY? We do in Ohio. Three failures within one hour last week. I stick to 500 GB drives.

    Edit: Yes, I compress when needed. Never had a problem. Windows XP pro sp3.

    John
     
    Last edited: Sep 6, 2012
  7. Sceadwian

    Sceadwian Banned

    Joined:
    Oct 27, 2006
    Messages:
    14,047
    Likes:
    141
    Location:
    Rochester, US
    There CAN be an advantage in specific situations under specific RAID levels, it does not mean that raid is superior to single drives, or practical in every situation, and for every price point.

    Raid mirror suffer from inefficiency of data storage, where stripped arrays suffer from fault tolerance, though stripped parity arrays are very common because they mitigate those main issues, they still suffer from the problems of both but to a lesser degree because of the integrated RAID chips nowdays.

    I've never found compression outside of off disk archiving to have any effect of the total load on storage. It's already been incorporated in audio/video codec's directly. There is virtually no data storage method outside of (and even including) text that doesn't have an integral compression layer already.
     
  8. Grossel

    Grossel Member

    Joined:
    Sep 16, 2008
    Messages:
    918
    Likes:
    22
    Location:
    Norway
    Hi.

    I tend to "compress" folders where it's contents is rarely used AND the type of contents is possible to actually compress.
    If the OS is installed to a ssd drive, then it is very interesting to make the size of the installed OS as low as possible.
     
  9. Sceadwian

    Sceadwian Banned

    Joined:
    Oct 27, 2006
    Messages:
    14,047
    Likes:
    141
    Location:
    Rochester, US
    Grossel, have you ever done any statistics on how much space you're saving with the compressed data?
     
  10. MrAl

    MrAl Well-Known Member Most Helpful Member

    Joined:
    Sep 7, 2008
    Messages:
    11,028
    Likes:
    951
    Location:
    NJ
    Hello,


    I prefer to use zip files when i really want something compressed. I feel that it should be me that decides if i want something compressed or not, not the operating system. There has to be a slowdown because all the data has to run through a program before you can access it.

    I've done quite a bit of testing on compression and even developed my own algorithms just to see how well i could get it to work. But the most important thing i have learned is that not all files compress as well as we would like them too, and there is always a compression ratio which is the new data size divided by the old data size. For example a file 10MB long that is compressed to 1MB has a 10 to 1 compression ratio, or simply 1/10 (because the new file is one tenth the size of the old file).
    Files that do compress well are text files and bitmap files (such as .bmp files). Jpg files for example will only compress a little more with one run through the compression algorithm and to me it's just not worth the time to get a compression ratio of 98/100. Text files on the other hand compress very well usually like to 1/3 original size.

    But there is one other reason for zipping files (and not compressing he whole drive). With the new SATA standards (this may improve in the future) and the standards on many run of the mill USB drives, it can take many many times longer to store 1000 1k files than it takes to store one single 1MB file which is a file with all those 1k files zipped up. In fact, you dont even have to zip them you can use a file zipping technique that simply puts all of the files in one big file end to end. Either way it takes MUCH less time to store the large single file than it takes to store 1000 of the smaller ones even though the data content is exactly the same. I thought i would mention this because it comes into play when you have to back up a lot of smaller files like 1k to 300k.
    The SATA standard is up to 6GBit/sec now (3GBit/sec not long ago) and that should give us at least 300MB/sec, but because of other problems (like head seeks) it is common to find the speed down much much lower when the drive is used in IDE mode, but i heard that AHCI mode isnt that much better.
    The newer 40MB/sec USB drives are better of course but they are also a LOT more money.
     
  11. Grossel

    Grossel Member

    Joined:
    Sep 16, 2008
    Messages:
    918
    Likes:
    22
    Location:
    Norway
    No, why reinvent the weel? There's a lot of test results on the web. However, being able to comress files transparently for the user is a nice feature, so why not take advantage of using it?


    And I do love 7z files for they ability to shrink physical data size even more :D
     
  12. Sceadwian

    Sceadwian Banned

    Joined:
    Oct 27, 2006
    Messages:
    14,047
    Likes:
    141
    Location:
    Rochester, US
    Cost/benefit. It slows down access time, it uses processor cycles. Disk space is so cheap that compression is effectively pointless unless you're talking about very very large databases, it's pointless to the 'end user' of a typical PC, and even to higher end users, there simply is no advantage.
     

Share This Page