Thoughts on External Sata for Fast Data Transfers

[MrAl]

Hello there,

I *could* start by griping about the current state of USB and data transfers .............. so i *will*

To start, USB data transfers depend a lot on the system and USB drive itself. I've tried a lot of different options, included USB drives that are advertised as being very fast (150MB/s) and i find some very very serious problems with USB, even USB 3.0 which was the current standard up to maybe several months ago when USB 3.1 came about. I found that the specifications on USB 3.0 are very very misleading and really absolute trash talk really. I'll elaborate here.

First, USB 3.0 can transfer large files pretty fast if the USB drive can handle it. That's not too bad, except for people demanding top speed, and then we find that the 500 megabytes per second quote is not true at all, and with overhead and all it comes down as low as 300MB/s and maybe you'll see 350MB per second if lucky and have the right drive, and that's probably with UASP enabled (another relatively new USB standard).

But for me, 300MB/sec would be just fine. The problem is, that's only with larger files. If we transfer one 1 gigabyte file we might be able to do it in 4 seconds, which isnt bad at all (i clocked 5 to 6 seconds on one of my drives). However, if we transfer that same amount of data but they are a bunch of small files, the transfer rate falls AT LEAST by 10 fold. So for that same 1GB data if that was 10000 files that were 100k each, the transfer rate could fall as low as 5 megabytes per second, but more typically 10 to 20MB/sec, and that's with a very very fast USB drive of the SSD type. I've read that 12MB/sec is the norm here.
So what happened? From 300MB/sec to 12MB/sec? That's just nuts. Somebody messed up somewhere.

I also tried using a Sata to USB converter cable/power supply with standard SSD, but it topped out at 20MB/sec. To be fair though, the converter was made for USB 2.0 and i didnt feel like buying a new one for USB 3.0 because i am afraid that the performance will still be too low.

Ok so i am done complaining, at least for a moment because i will never stop complaining about this one

To move on, i thought about using an external Sata drive. the external drive would plug into the mother board eSata port or a PCIe card could be purchased and plug the drive into that. Using a good regular SSD drive, we could easily see transfer rates of 400 megabytes per second, and depending on the IO transfer rate of the drive, probably 100MB/sec for groups of small files. 100MB/sec isnt too bad really, and i could live with that. It's just hard to live with 10MB/sec when the drive itself is much much faster than that.

So to sum up, using a USB SSD drive i can get normally probably 15MB/sec most of the time, but using an external Sata can probably get 100MB/sec most of the time, so i might move to using eSata instead. The Sata SSD drives are very light weight and easy to carry in a pocket, although not as nice as a regular USB drive.

I do have reserves about this though. The main worry is about the Sata and Power connectors on the SSD drive. I dont think they are made for thousands of insertions and removals. I could be wrong because i have seen 'docking' stations that quote 50000 insertions but i dont know how accurate that figure is.
So in short, i wonder how many insertions the drive would take before it started to fail now and then.
I know i could make up an adapter, and that would put the burden on the adapter, which could be changed periodically, but, that would make the drive harder to carry around (it has to be portable).

A secondary issue is that the external drive might read at 15MB/sec for small files the first time they are read, but then what is strange is the second time they are read they read at 150MB/sec !! Is that strange or what?

Any ideas on any of these topics?

To recap the topics were:
1. Sata connector insertion issue.
2. Read once slow, read second time fast.

Any ideas would help as i have been wrestling with these issues for several years now. I am about to do something drastic but it would take a lot of work and time so i would like simpler ideas.

 

[MrAl]

Hi,

No thoughts on this?

I remembered i forgot to mention one other issue too which is strange.
While copying files on my 'regular' USB 3.0 thumb drive rated for about 50Megabytes per second, it will copy files at a rate of maybe 5MB/sec to start with, and after copying around 100 files it will stop for a few seconds, then start again.
Also, sometimes it will stop for several seconds, like as much as 10 seconds, then start again.
This tells me that USB flash drives may not be made consistently, with 'bad' spots in the memory chips in some locations, causing them to take much much longer to write to those areas. After it stops for maybe 10 seconds it will start again and then seem ok for a while again.

This also happens with one of my standard hard disk Sata drives too, although it seems rare to hit a bad spot. When i copy a lot of data i see this happen sometimes. I think it is partly due to the disk drivers for Windows, because i think if the write does not go well it keeps trying and trying until it does go well, and that could take a long time with a weak sector. In the old days of Windows 3.1 we could lock out that sector but these days they just keep trying and trying over and over in the same spot until it gets written or the user gets tired of waiting and hits 'cancel' or something like that. That should not be the way it is done.
In the past i have been able to manually lock out sectors, but it's not easy to do because i have to wait to find the problem section and then make a note to never erase the files that had just been copied, that way that part of the disk never gets written to again. And of course that means no disk defrag because that could move the files and 'free' up that bad section again.

So disk storage still seems to have a long way to go yet. The biggest issue i read complaints about is the small file copy rate, which depends highly on the I/O rate of the disk not the sequential sustained Megabytes per second rating. If the I/O rate is low, so is the small file copy rate so if you have a lot of small files to copy you have a nice long wait on your hands even with a good disk.

There are ways around this but they make the file system more complicated, and apparently nobody implements anything like this yet.
I've read that AS Rock has software that 'helps' the USB transfers, but from the test numbers i see they still havent hit the nail on the head yet. Modern multi core systems can do much much better with a little driver update. I guess we have to wait for them to get around to addressing this issue in a serious way.