How to manage removable disk drives?

There have been a number of issues raised recently regarding USB flash drives.
Could someone give definitive answers to the following

  • is it safe to unplug a drive while the computer is powered on?
  • should a drive be unmounted before removal?
  • should sync be used before unmounting or power off?
  • should sync be used after writing to a usb drive?
  • what if the computer is rebooted rather than powered off?
  • what if one uses halt or shutdown
  • what if the computer is in suspend or hibernate?
  • what if a drive is used on Win as well as Linux?

I must admit I have always been super conservative and never remove a drive until the computer is completely powered off… But they are supposed to be plug-n-play arent they?

1 Like

Hi Neville, :wave:

I must admit I may not be knwledgeable enough to answer all of your questions. :blush:

However that´s the way I do it:

If it for any reason might be necessary to disconnect a USB flash drive I don´t think you must power down the system beforehand.
Using thunar as my daily file manager there´s the option “eject” when right-clicking on the
left-hand side (“devices”).
So I just eject the device and then - I guess - it´s safe to remove it.

Before shutting down my system I do the same. No idea if that´s totally neccessary; I just do it anyway. :blush:

Hmm, I think that question should be reserved for someone elso to answer. I´ve been on Linux/Lubuntu for about six years now … exclusively.

As far as reboots are concerned I never bother with ejecting or unmounting them first.
Suspend and hibernate I never use.

So hopefully some other folks may be able to shed some light on the matter.

Many greetings from Rosika. :slightly_smiling_face:

1 Like
  1. Yes, if not in use.
  2. Yes, because it must not be in use.
  3. No need. It’s usually just about being paranoid.
  4. No need. It’s usually just about being paranoid.
  5. Depends if “reboot” is “enhanced” to not make it a real reboot anymore, as it is the default in Windows.
  6. That turns off the computer.
  7. Suspend and Hibernate need to be set up properly in Linux, to work properly. Otherwise, these modes may cause issues.
  8. Not a great idea, but doable. It’s just that you would need to choose the lowest common denominator between the two, which is some FAT32 or NTFS file system, which in turn means you cannot do anything on Linux with files on these file systems, except read, write & copy/move them.
2 Likes

Never thought of using eject.
I thought it only applied to DVD’s
Does eject do an unmount? I do not know . Must check

Thank you Rosika
Neville

1 Like

@Akito
I disagree with your No 4 answer
When I dd a large file to a usb drive at the command line, the dd returns a message saying ‘x’ bytes read and ‘y’ bytes written, and ‘x’ is always much larger than ‘y’, so it has a huge number of bytes still in a memory buffer… and it returns control to the user
So I issue a sync
and then it hangs for ages while it flushes the buffer, before returning a prompt.
Not sure what would happen if I tried an unmount instead of a `sync’… never been game enought to try it

To me, dd should not be returning control to the user, before the buffer is flushed?

More questions:
Are memory cards the same?
Is @Rosika 's use of eject the same as unmount?

And thank you,
it makes more sense to me now, and hopefully to others
Neville

2 Likes

That Linux is badly designed then. It shouldn’t be ever necessary to sync manually. Nowadays, it’s usually a sign that something’s wrong, if it’s truly necessary. Making the user watch a memory buffer is a UX nightmare.

Indeed. This should be default behaviour. The problem with dd is, it’s a truly ancient tool. I wouldn’t recommend using it. If someone really wants a low-level dd-like tool, he should use ddrescue instead. It’s like dd but it actually is not a pain in the ass to use, as dd is.

2 Likes

It is not only dd, it happens with rsync too.

I dont think it is necessary to sync.
I think if you waited, the OS would sync the buffers in its own time.
Trouble is you dont know how long to wait.

Floppy disks used to behave like this all the time, even with a simple cp.

2 Likes

Hi Rosika,
I tested eject in Thunar.
You are right, it unmounts the USB drive.
So unplugging it should be safe… except I am not sure what would happen if you had written to the USB drive and it had not finished flushing the buffer when you ejected. I would hope eject would sync the buffer before unmounting. ( see discussion with @Akito about dd and sync)
Regards
Neville

1 Like

Overkill… if I’ve plonked data on there - I run a sync (especiall “sudo sync” if I’d been root doing a “dd” or something), sometimes I eject (CLI or nautilus)… Mostly I don’t… So long as sync finished I’m happy…

I’ve only ever had dramas with removable media on Windows… I don’t run or use Windows anywhere these days… Hope to never go back there again…

I have heaps of Fat32 USB drives around - I plonk mp3 files on them for when I’m driving the car… I’m slowly replacing my MP3 collection with FLAC, but the USB music player in the car (10 year old Toyota) won’t play anything but mp3, or see anything but Fat32… Never had any problems there either - I strongly suspect there’s a Linux system behind the ugly UI on the Toyota music screen (that’s all it does - plays music, or interfaces with a phone - but don’t get me started on that - the Bluetooth on it is so shonky it can take 15 minutes to pair! 10 year old LINUX was GARBAGE for Bluetooth!).

Anyway - I insert and remove USB so often a full power cycle would be a huge PITA - also - considering I barely EVER shutdown or reboot my main desktop, I don’t plan on changing that…

Well it is - it inherited this from UNIX… I can remember running sync on Solaris all the time, not necessarily after dd, but I just got use to doing it… Remember “dd” is an old UNIX tool… I used to use dd to create TAR files from raw TAR (TApe aRchive) images off floppy disks… more reliable than direct read tar from the device…


sync; sleep 5 ; sync ; uadmin 2 0

“to be sure, to be sure”

2 Likes

I was not advocating everybody do shutdowns to remove a Usb drive. Its just easy for me because I always shutown when finished for the day. Your usage patterns are different.

I agree sync is necessary… because Unix utilities like dd or rsync or even cp, return control to the user, without telling you if or when the buffer is flushed.
That behaviour is fine for an HD, but not for removable devices.
This is something Linux/Unix need to fix.

1 Like

Hi Neville, :wave:

Thanks for testing it. You were quicker than me. :smiley:

I´ve never tried this before but I may tell you the following:

When I change into the USB-stick-directory in thunar and open a terminal with “open terminal here” then the teminal-related process is connected to the (mounted) USB-stick, right?
Because the terminal-command pwd shows me e.g.

rosika@rosika-10159 /m/r/74C1-30C7> pwd
/media/rosika/74C1-30C7

If I don´t close the terminal (and haven´t changed the directory) and then try to eject the USB-drive from within thunar it won´t let me by default.
A popup message appears saying something like “there´s still data being written to the … drive”.

I guess that message would also appear if any other process like dd was still making use of the drive.

… at least that´s my experience with thunar. Hope it helps.

Many greetings.
Rosika :slightly_smiling_face:

2 Likes

Hi Rosika,
Yes, if you look at the umount or eject man pages, it says that umount or eject will not proceed if the device is ‘busy’, which it clearly was in your case because your terminal was ‘in’ the devices directory.

What I would like to know is what does ‘busy’ mean. In particular does it cover the case where you were writing some files to the usb drive, the write process terminated , but the buffers were still being flushed?

I guess that message would also appear if any other process like `dd` was still making use of the drive.

Yes, if dd were still running, that message would appear.
But the issue is that dd stops running, and returns control to the user, before the the write process is completed. It relies on the system sync to complete the emptying of the buffers.

We shall find the answer, but it is not clear in the man pages
Regards
Neville

1 Like

Hi Neville, :wave:

O.K., I see now what you mean.
Sorry to have stated facts that were already obvious to you. :neutral_face:

I was looking around a bit and found on usb - Disable Nautilus caching mechanism - Ask Ubuntu the following statement by vidarlo :

In addition, Ubuntu will not tell you that it’s safe to remove the device before the cache is flushed.

Hmm, I guess we´d have to look a bit further… :thinking:

Many greetings from Rosika :slightly_smiling_face:

from the source mentioned above:

What you see in action is the file buffer. First, content is copied to the file buffer. When the buffer is full, copying continues to the file buffer, while on the other end, content is physically being flushed to the disk.

Once copying is “finished” for the file manager, the buffer is still being flushed in the background until complete. This is how Linux works, and it will work the same with nautilus Files as with any other file manager on linux.

That you have gotten corrupt files is not the fault of Canonical. It is your fault. To correctly unmount a volume, you need to click the “eject” button in the file manager, or right-click and select “Safely unmount disk”.
If writing is still ongoing, you will get a notification that you should not yet unplug the drive. Once it is safe to eject the drive, you also will get a notification.

So await the notification before removing the drive. No software or instrumentation ever is 100% foolproof against unproper use.

As I indicated in my answer, you do not need to guess. You just need to unmount properly using software: then the system informs you. Also on a server a device must be properly unmounted. If using the terminal, the system will tell you it cannot if writing still is ongoing.

Hi Rosika,
No,no. Nothing is obvious. We need the facts stated

In addition, Ubuntu will not tell you that it’s safe to remove the device before the cache is flushed.

Well that is the bad news that I feared
If that is correct, then eject or umount will not protect the user from unmounting before the buffer is flushed

There is only one way to be sure… I will get a spare usb drive, write something huge like an iso file to it, then when dd returns control to the user, immediately try to eject or umount it. We shall see

Regards
Neville

1 Like

Hi again, :wave:

On usb - Power off vs. "safely eject" or "unmount" of external devices (cameras, (android) smartphones, externally powered hdd's, etc)? - Ask Ubuntu I found this:

The point behind “unmounting” or “safely removing” is that you have to make sure all the buffered data has been flushed ( or synced with the device) .

So if you just poweroff the device via their operating systems (i.e soft poweroff ) , it would be fine because all operating systems make sure all I/O operations are done before actually shutting down the device.
But if you mean a hard poweroff (e.g plugging out a HDD ) , it might cause data corruption or even in the worst case , filesystem corruption if you do this when a change in the partition table is being made

1 Like

Perhaps for testing… :question:

If you’re worried about data loss, you could create a new blank file , write a single character into it, save and then immediately press eject.
After remounting, if the file is there as well as the character you typed, then it is syncing properly before unmounting.
A single character would definitely stay in the buffer so that’s a good test. The eject will also fail if the drive is being used to avoid corruption.
In terminal, cd to your removable drive, type while true; do ls; done and then try to eject the disk. It should say it’s busy.

(source: mount - Can you unmount Drive in Ubuntu's GUI? - Ask Ubuntu )

1 Like

Well I did the above experiment, except used rsync instead of dd.
Here is what happened

# rsync -aAXvH --exclude=Downloads/li*  ./*  /mnt/TestData
sending incremental file list
Downloads/
Downloads/tsetup.3.7.3.tar.xz
Downloads/waterfox-G4.1.1.1.en-US.linux-x86_64.tar.bz2
Downloads/waterfox-G4.1.2.en-US.linux-x86_64.tar.bz2

sent 207,566,275 bytes  received 79 bytes  59,304,672.57 bytes/sec
total size is 229,587,244  speedup is 1.11
# eject /dev/sdb1
     ...... this ran for about a minute before returning the prompt  ...
# sync
     .....   this returned the prompt immediately......
# 

Look at the difference between sent and received at the end of the rsync… there is a lot of bytes still in the buffer/cache
I typed eject immediately … it hung for over a minute before returning the prompt — ie it was flushing the buffer before completing the eject.
The sync after the eject returned immediately … ie there was nothing left to flush.

Normally, if I typed sync immediately after the rsync finished, it would also take some time before returning.

Therefore either sync or eject will flush the buffer.

I did the whole thing again with umount instead of eject, and same result … umount waits until the buffer is flushed before returning.

Conclusion:
If you copy any material to a removable drive ( with dd or rsync or cp or some other utility) , it is likely that the copy routine will return the user’s prompt, before the buffer/cache is fully flushed. Therefore you should use one of eject, umount or sync, and wait for it to return the prompt, before attempting to unplug the drive.
So @Rosika was OK using eject and what @daniel.m.tripp does with sync is also OK. What @Akito said about sync not being necessary is also right, provided one uses umount or eject before unplugging the drive.

Just a note on the difference

  • sync flushes the buffer(s) but leaves the drive mounted and present in Thunar or as an icon
  • umount flushes the buffer and then unmounts the drive filesystem, so it will not appear in a df table, but leaves the device present in Thunar and as an icon
  • eject flushes the buffer and then unmounts the filesystem so it will not appear in a df table, and then also removes the device so it will not appear in Thunar or as an icon. In some cases eject requires umount first.

What happens if you make a mistake and do a umount or eject while the copy process is still running? It depends who you are. For a normal user, it fails, for superuser it kills the copy process so it does not complete. So dont make that mistake!

So this answers the original questions 2 and 3.
You should use umount or eject before unplugging a drive after a write
You should use sync after a write , if you plan to read back material from the drive.
And in both cases you should wait for the eject or umount or eject to complete before proceeding.

Thanks to @Rosika and @daniel.m.tripp and @Akito
Regards
Neville

1 Like

Hi Neville, :wave:

thank you so much for putting a lot of time and effort into solving the problem … and above all, letting us know what you came up with. That´s really great. :heart:

I´m sure there are some people who may benefit from your findings. At least that´s certainly true for me. :blush:

Thanks for confirming that.

Yes, that´s really great. I´ll make a note of that in my how-to-folder . :+1:

Thanks once more for this great work of yours.
Many greetings from Rosika :slightly_smiling_face:

1 Like

Using Linux Mint ,I just plug-in or remove any USB flash stick without any other operation . In both cases I just hear a THUMB and that is it.
Sofar I never damaged the stick or crippled any file on it

Frank in County Wicklow - Ireland