Testing read and write speed of a device

Rosika · June 5, 2023, 3:06pm

Hi all,

I have a question about testing the speed of writing to a device.
To be more exact: It´s a new usb-stick of mine which I want to test.

Is there a terminal command (or a package to be installed) with the help of which I can test the speed

I´m particularly interested in the speed of writing to the device as I suspect it´s a bit slow.
Well, the stick itself is connected to a USB 2.0 port and therefore it should be clear that the speed of writing isn´t all too fast.

Still: I´d like to get the exact values, if at all possible, in order to be able to compare them.

The thing however is: There´s already quite a lot of data on the stick that I don´t want to lose.
So if any command for my purpose would need the whole of the stick or even deletes all of its contents, this wouldn´t be the thing I´m looking for.

Basically the command should write to a dedicated part of the stick only (e.g. a special folder) to provide me with the info I´d like to get.

Does anyone know if there´s a command than can do that

Many thanks in advance.
Rosika

Rosika · June 5, 2023, 4:38pm

I´ve done a bit of research and found this source:

Well, I have to admit ChatGPT pointed me in the (hopefully) right direction.

I still have to investigate and verify the commands, but this could be a possible solution:

fallocate -l 1G /path/to/your/usb/mount-point/testfile # create a file of a specific size (e.g., 1GB) on my USB stick:
fio --name=write_test --ioengine=sync --rw=write --bs=4k --numjobs=1 --size=1G --time_based --runtime=60s --filename=/path/to/your/usb/mount-point/testfile

ChatGPT says:

This command will write 1GB of data to the specified file for 60 seconds and measure the write speed. You can adjust the parameters as needed to fit your requirements.

Note that the fio command doesn’t delete any existing data on your USB stick. It only writes to the specified file for the duration of the test.

Once the test completes, you’ll receive the results, including the write speed. This will give you an idea of the write performance of your USB stick.

…which is basically what I´m looking for.

Before running the fio command I will have to check the parameters suggested first.
I have to admit I´ve never came across fio before, so it´s all new to me.

I´ll look into it and post the outcome in due course.

Many greetings
Rosika

nevj · June 6, 2023, 12:08am

Do a backup first

Rosika · June 6, 2023, 11:44am

Thanks, Neville, for the recommendation .

In fact doing a backup of the 3 usb-sticks, which are constantly attached to my system, is part of my monthly backup-routine.

My main system I backup with clonezilla.

Many greetings
Rosika

Rosika · June 6, 2023, 2:39pm

Hi again,

after doing some research on fio´s command options it seemed to me the syntax provided by ChatGPT (my post #2) is correct.

So I first created a 1GB file on the stick (using the fallocate command) and then issued the respective fio command. It took about 3 or 4 minutes to run and here´s what fio came up with:

fio --name=write_test --ioengine=sync --rw=write --bs=4k --numjobs=1 --size=1G --time_based --runtime=60s --filename=/media/rosika/0002-2D2B/test_für_fio/testfile 
write_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]                          
write_test: (groupid=0, jobs=1): err= 0: pid=23911: Tue Jun  6 15:58:29 2023
  write: IOPS=3498, BW=13.7MiB/s (14.3MB/s)(820MiB/60001msec); 0 zone resets
    clat (usec): min=3, max=45931, avg=275.54, stdev=1984.71
     lat (usec): min=5, max=45932, avg=277.49, stdev=1984.72
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   11], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   13], 90.00th=[   16], 95.00th=[   21],
     | 99.00th=[15270], 99.50th=[15270], 99.90th=[19268], 99.95th=[19268],
     | 99.99th=[22676]
   bw (  KiB/s): min= 6912, max=273456, per=100.00%, avg=14050.02, stdev=33442.21, samples=119
   iops        : min= 1728, max=68364, avg=3512.50, stdev=8360.55, samples=119
  lat (usec)   : 4=0.01%, 10=42.27%, 20=52.72%, 50=3.17%, 100=0.02%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 10=0.01%, 20=1.78%, 50=0.02%
  cpu          : usr=0.99%, sys=6.45%, ctx=3876, majf=1, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,209917,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=13.7MiB/s (14.3MB/s), 13.7MiB/s-13.7MiB/s (14.3MB/s-14.3MB/s), io=820MiB (860MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/4416, merge=0/5, ticks=0/58648, in_queue=58648, util=98.12%
rosika@rosika-Lenovo-H520e /m/r/0/test_für_fio> echo $status
0

Hmm…

it looks a bit overwhelming to me. Much information, which is a good thing… but where exactly do I look for the results of the write speed test

Does anyone have a clue?

Thanks and many greetings from Rosika

P.S.:

my guess is: this line could be relevant

easyt50 · June 6, 2023, 6:26pm

My thought exactly! Except I believe it is too much information. What you wanted was the average read / write speed.

Have not tried it, but this web page looks interesting. It shows how to “Measure Disk Performance with fio,dd and Graphical Method.”

Kind Regard,
Howard

easyt50 · June 6, 2023, 9:14pm

Hi Rosika,

I found 2 others things that will affect how fast your PC can read / write to the USB Flash (Stick) drive.

USB 2.0 ports.
“USB 2.0 clock speed is 480 megabits per second. That’s 60 megabytes per second. Given the protocol overhead and the fact that USB 2.0 is half-duplex, the maximum data rate will be 30-40 megabytes per second. The 480 megabits per second limit applies to the USB controller and is shared between the ports attached to it. The number of USB controllers per card or motherboard will vary.”

And I found this chart showing the max speed of USB flash drives. The flash drive should either show this code or it should be shown in the packaging.

don.karon · June 6, 2023, 10:36pm

In Ubuntu open the app “Disks” > click on the disk you want to check > click on the three vertical dots in the upper right corner of the window > choose “Benchmark disk”.

daniel.m.tripp · June 7, 2023, 12:21am

There was also this thread from 3 years ago - more about SD-Cards, but still relevant to USB thumb drives :

nevj · June 7, 2023, 12:42am

So 14 Mb/s
How do you format your usb stick?
The type of filesystem could make a difference.
You probably should use ext for Linux

daniel.m.tripp · June 7, 2023, 5:50am

I’m just now facing the opposite issue…

I paid paragon software what seems an exhorbitant ($40+ is not cheap) license fee to use their ext4 “driver” on MacOS (and it will only let me use it on ONE COMPUTER!) so I could share ext4 thumb drives with my Linux machines…

Just hit a wall - upgraded my personal MacBook Pro to MacOS 13.4 (from 12.62) and it broke that piece of crap driver and I lost my Resilio Sync share on it…

Reformatted as APFS (encrypted) now and having to sync ~200 GB of data over the network (NFS on ethernet)… IF I need to get files on it, or off it - I’ll just use scp or rsync or something…

So a big FU to Paragon Software! Thanks for nuthin’!

Also - I also updated my work issued MBP (M1) to Ventura earlier - and didn’t hit this issue - because I wasn’t using ext4… The only reason I use a thumb drive on my personal MBP is it only has 256 GB SSD - so I offload my bigger Resilio Sync “sync targets/sources” onto external storage (I have a FAST USB C, USB 3.2 thumb drive for this purpose)…
If I can’t use SCP / rsync to copy data, I can just use a Fat32 USB drive… I’ve like over 30 of them 8 GB or over… My Ventoy stick is visible on the MacBook (not that I’d try to boot it - I’m quite happy with Mac OS X / UNIX - don’t need Linux on arm64 / M1 / Apple Silicon thank you very much).

nevj · June 7, 2023, 11:57am

While we are on USB sticks, I gambled $10 on a 1Tb stick. It came with instructions in some foreign script. It is only usb2.0. It does work… at least i plugged it in and the computer could see its filesystem.
How do you reckon it will perform? What is the life expectancy ?
I might try Rosika’s fio speed test.

Rosika · June 7, 2023, 1:20pm

Hi all,

thank you so much for your feedbacks and replires.

@easyt50 :

Yes, Howard. fio focusing on the relevant information would´ve been nice.

Actually I just wanted to know what the write speed was.
Thanks also for the link.

Funny thing though: after giving an example for fio they say:

The results are as follows: […]

… and provide the output for the example command, but they don´t seem to explain it.

As for the other methods: I´ll have to read the article in its completeness, of course.

Thanks, Howard, for the additional info.

Yes, I have just USB 2.0 ports available on my system. Still: write speed for the USB-stick should´ve been higher than what I have experienced (see: below).

The chart you provided looks intersting. Thanks for digging it out.

@don.karon :

Thanks, Don, for the hint.

I found “Benchmark disk” in the application. I have to do some research on it first as I don´t know whether this option would destroy any data on the stick or whether it works in a similar way as fio does.

@daniel.m.tripp :

Thank you, Dan, for providing the link to the older thread.
I´ll have a good look at it.

@nevj :

Thanks, Neville, for the confirmation.
That was my guess, too, but I wanted to be sure.

I see. Well, I hadn´t thought of that.

The stick in question is a “Verbatim STORE N GO (PMAP)” model.
I originally purchased it as a storage medium for recording programmes with my DVB-T2 receiver (terrestial digital TV).
It proved to be less than optimal for my purposes, so I decided to use it with my PC instead.

It was already formatted in FAT32 and I also compared it to the other USB-sticks I have attached to the PC and they are FAT32 as well. So I just left the formatting as it was.

The reason for getting interested in the write performance of the stick is:

I was copying data from another USB-stick (but also from the external HDD) to this Verbatim-stick.
I was using rsync for that:

Fortunately I took a screenshot of the terminal at the time to see what the different write speeds were. I marked them in red:

I was copying the same data…

to the Verbatim stick: 218.14 kB/s
to the HDD (# also connected via USB 2.0) : 7.98 MB/s

That sparked my interest in knowing more about writing speeds.

But:

as these are results of the fio writing speed test:

… why would my copy action only produce 218 Kb/s

It´s a mystery to me… .

Thanks so much to all of you and many greetings from Rosika

don.karon · June 7, 2023, 1:45pm

Rosicka -

The disk benchmark utility has never corrupted the data on my disks.

Don

Rosika · June 7, 2023, 1:55pm

Thanks, Don, for the confirmation.

Looks good then. I thought it might be wise to ask first.

Cheers from Rosika

nevj · June 7, 2023, 10:40pm

Hi Rosika,
Good question.
Copying a large number of small files is more work than copying fewer large files.
If there is already something on the disk, rsync has to do reads as well as writes.
Rsync uses the cpu, if the computer is busy it will share time between the copying and other jobs.
What are those rsync errors?
You are reading one usb stick and writing another. The usb data bus may be very busy.

It still seems rather a large difference to me.?

Regards
Neville

Try mounting it with a mount statement and setting async

Rosika · June 8, 2023, 1:01pm

Hi Neville,

I see. Well, the terminal screenshot shows the difference between writing to the Verbatim-stick and to the HDD.
But it depicts the same scenario. So the exact same files were copied…

That´s right.
But the HDD (external HDD) is also connected via USB 2.0, the same way the Verbatim stick is.

Yes, that´s my view on the matter as well.

Hmm, good question.
I don´t know. Have to investigate…

Well, the copying process still worked, but incredibly slowly.

Thanks a lot and many greetings from Rosika

P.S.:

Thanks also for the link. I´ll look into it.

kovacslt · June 8, 2023, 1:58pm

Flash drives are way much slower for writing, than reading, whereas HDD’s perform the same for reading and writing (usually )

Recently I fill up 64GB thumbdrives with the same content, so did the fillup once, imaged that drive, and put the image onto the copies with dcfldd.
Filling up the ‘master’ drive took about 3 hours, copying the images take about 2 hours.
While doing the copy, sometimes watch the process with iostat, and I see transfer speeds about 12MB/s.
The drives are USB3 Kingston G3 (or Exodia, which I got cheaper). Reading from these drives are blazing fast, I get speed about 90MB/s.
So writing to a flashdrive is slower, maybe much slower than the speed specification of that thumbdrive.

Rosika · June 8, 2023, 2:40pm

@kovacslt :

Hi László,

Thanks for the information.

I´ve never heard of dcfldd before, so I took a look at it with apt-cache show dcfldd .
It looks very intersting indeed.

Your report of what you did and achieved is amazing. It clearly shows how much practical experience and knowledge you possess. It´s really amazing.

Thanks for your reply.

@nevj :

I just asked ChatGPT (to have a quick result) and this is what it had to say about it:

The error message you encountered while using the rsync command indicates that the process was terminated by a signal. Here’s the breakdown of the error message:
rsync error: received SIGINT, SIGTERM or SIGHUP (code 20) at io.c(518) [generator=3.2.7]
This part of the error message suggests that the rsync process received a signal, which could be one of the following:

SIGINT: This signal is sent when the user interrupts a process by pressing Ctrl+C in the terminal.

SIGTERM: This signal is typically sent to terminate a process.

SIGHUP: This signal is sent when the terminal session is disconnected or closed.
rsync error: received SIGINT, SIGTERM or SIGHUP (code 20) at rsync.c(713) [sender=3.2.7]
This part of the error message further specifies that the signal was received within the rsync sender code.

These signals usually indicate that the rsync process was interrupted or terminated prematurely.
It’s possible that the process was manually interrupted by pressing Ctrl+C or that some external factor caused the termination, such as a loss of connectivity or an issue with the source or destination drive.
To avoid such interruptions and ensure smoother copying, you can try the following: […]

Thanks a lot to all of you.

Many greetings from Rosika

kovacslt · June 8, 2023, 2:46pm

From my point of view it’s just dd. I needed a tool to write the same image to multiple drives at once. The dcfldd does this well
( This was my hurray post , when I found it: A use case of dcfldd )