I must admit I may not be knwledgeable enough to answer all of your questions.
However that´s the way I do it:
If it for any reason might be necessary to disconnect a USB flash drive I don´t think you must power down the system beforehand.
Using thunar as my daily file manager there´s the option “eject” when right-clicking on the
left-hand side (“devices”).
So I just eject the device and then - I guess - it´s safe to remove it.
Before shutting down my system I do the same. No idea if that´s totally neccessary; I just do it anyway.
Hmm, I think that question should be reserved for someone elso to answer. I´ve been on Linux/Lubuntu for about six years now … exclusively.
As far as reboots are concerned I never bother with ejecting or unmounting them first.
Suspend and hibernate I never use.
So hopefully some other folks may be able to shed some light on the matter.
Depends if “reboot” is “enhanced” to not make it a real reboot anymore, as it is the default in Windows.
That turns off the computer.
Suspend and Hibernate need to be set up properly in Linux, to work properly. Otherwise, these modes may cause issues.
Not a great idea, but doable. It’s just that you would need to choose the lowest common denominator between the two, which is some FAT32 or NTFS file system, which in turn means you cannot do anything on Linux with files on these file systems, except read, write & copy/move them.
I disagree with your No 4 answer
When I dd a large file to a usb drive at the command line, the dd returns a message saying ‘x’ bytes read and ‘y’ bytes written, and ‘x’ is always much larger than ‘y’, so it has a huge number of bytes still in a memory buffer… and it returns control to the user
So I issue a sync
and then it hangs for ages while it flushes the buffer, before returning a prompt.
Not sure what would happen if I tried an unmount instead of a `sync’… never been game enought to try it
To me, dd should not be returning control to the user, before the buffer is flushed?
Are memory cards the same?
Is @Rosika 's use of eject the same as unmount?
And thank you,
it makes more sense to me now, and hopefully to others
That Linux is badly designed then. It shouldn’t be ever necessary to sync manually. Nowadays, it’s usually a sign that something’s wrong, if it’s truly necessary. Making the user watch a memory buffer is a UX nightmare.
Indeed. This should be default behaviour. The problem with dd is, it’s a truly ancient tool. I wouldn’t recommend using it. If someone really wants a low-level dd-like tool, he should use ddrescue instead. It’s like dd but it actually is not a pain in the ass to use, as dd is.
I tested eject in Thunar.
You are right, it unmounts the USB drive.
So unplugging it should be safe… except I am not sure what would happen if you had written to the USB drive and it had not finished flushing the buffer when you ejected. I would hope eject would sync the buffer before unmounting. ( see discussion with @Akito about dd and sync)
Overkill… if I’ve plonked data on there - I run a sync (especiall “sudo sync” if I’d been root doing a “dd” or something), sometimes I eject (CLI or nautilus)… Mostly I don’t… So long as sync finished I’m happy…
I’ve only ever had dramas with removable media on Windows… I don’t run or use Windows anywhere these days… Hope to never go back there again…
I have heaps of Fat32 USB drives around - I plonk mp3 files on them for when I’m driving the car… I’m slowly replacing my MP3 collection with FLAC, but the USB music player in the car (10 year old Toyota) won’t play anything but mp3, or see anything but Fat32… Never had any problems there either - I strongly suspect there’s a Linux system behind the ugly UI on the Toyota music screen (that’s all it does - plays music, or interfaces with a phone - but don’t get me started on that - the Bluetooth on it is so shonky it can take 15 minutes to pair! 10 year old LINUX was GARBAGE for Bluetooth!).
Anyway - I insert and remove USB so often a full power cycle would be a huge PITA - also - considering I barely EVER shutdown or reboot my main desktop, I don’t plan on changing that…
Well it is - it inherited this from UNIX… I can remember running sync on Solaris all the time, not necessarily after dd, but I just got use to doing it… Remember “dd” is an old UNIX tool… I used to use dd to create TAR files from raw TAR (TApe aRchive) images off floppy disks… more reliable than direct read tar from the device…
I was not advocating everybody do shutdowns to remove a Usb drive. Its just easy for me because I always shutown when finished for the day. Your usage patterns are different.
I agree sync is necessary… because Unix utilities like dd or rsync or even cp, return control to the user, without telling you if or when the buffer is flushed.
That behaviour is fine for an HD, but not for removable devices.
This is something Linux/Unix need to fix.
I´ve never tried this before but I may tell you the following:
When I change into the USB-stick-directory in thunar and open a terminal with “open terminal here” then the teminal-related process is connected to the (mounted) USB-stick, right?
Because the terminal-command pwd shows me e.g.
If I don´t close the terminal (and haven´t changed the directory) and then try to eject the USB-drive from within thunar it won´t let me by default.
A popup message appears saying something like “there´s still data being written to the … drive”.
I guess that message would also appear if any other process like dd was still making use of the drive.
… at least that´s my experience with thunar. Hope it helps.
Yes, if you look at the umount or eject man pages, it says that umount or eject will not proceed if the device is ‘busy’, which it clearly was in your case because your terminal was ‘in’ the devices directory.
What I would like to know is what does ‘busy’ mean. In particular does it cover the case where you were writing some files to the usb drive, the write process terminated , but the buffers were still being flushed?
I guess that message would also appear if any other process like `dd` was still making use of the drive.
Yes, if dd were still running, that message would appear.
But the issue is that dd stops running, and returns control to the user, before the the write process is completed. It relies on the system sync to complete the emptying of the buffers.
We shall find the answer, but it is not clear in the man pages
What you see in action is the file buffer. First, content is copied to the file buffer. When the buffer is full, copying continues to the file buffer, while on the other end, content is physically being flushed to the disk.
Once copying is “finished” for the file manager, the buffer is still being flushed in the background until complete. This is how Linux works, and it will work the same with nautilus Files as with any other file manager on linux.
That you have gotten corrupt files is not the fault of Canonical. It is your fault. To correctly unmount a volume, you need to click the “eject” button in the file manager, or right-click and select “Safely unmount disk”.
If writing is still ongoing, you will get a notification that you should not yet unplug the drive. Once it is safe to eject the drive, you also will get a notification.
So await the notification before removing the drive. No software or instrumentation ever is 100% foolproof against unproper use.
As I indicated in my answer, you do not need to guess. You just need to unmount properly using software: then the system informs you. Also on a server a device must be properly unmounted. If using the terminal, the system will tell you it cannot if writing still is ongoing.
No,no. Nothing is obvious. We need the facts stated
In addition, Ubuntu will not tell you that it’s safe to remove the device before the cache is flushed.
Well that is the bad news that I feared
If that is correct, then eject or umount will not protect the user from unmounting before the buffer is flushed
There is only one way to be sure… I will get a spare usb drive, write something huge like an iso file to it, then when dd returns control to the user, immediately try to eject or umount it. We shall see
The point behind “unmounting” or “safely removing” is that you have to make sure all the buffered data has been flushed ( or synced with the device) .
So if you just poweroff the device via their operating systems (i.e soft poweroff ) , it would be fine because all operating systems make sure all I/O operations are done before actually shutting down the device.
But if you mean a hard poweroff (e.g plugging out a HDD ) , it might cause data corruption or even in the worst case , filesystem corruption if you do this when a change in the partition table is being made
If you’re worried about data loss, you could create a new blank file , write a single character into it, save and then immediately press eject.
After remounting, if the file is there as well as the character you typed, then it is syncing properly before unmounting.
A single character would definitely stay in the buffer so that’s a good test. The eject will also fail if the drive is being used to avoid corruption.
In terminal, cd to your removable drive, type while true; do ls; done and then try to eject the disk. It should say it’s busy.
Well I did the above experiment, except used rsync instead of dd.
Here is what happened
# rsync -aAXvH --exclude=Downloads/li* ./* /mnt/TestData
sending incremental file list
sent 207,566,275 bytes received 79 bytes 59,304,672.57 bytes/sec
total size is 229,587,244 speedup is 1.11
# eject /dev/sdb1
...... this ran for about a minute before returning the prompt ...
..... this returned the prompt immediately......
Look at the difference between sent and received at the end of the rsync… there is a lot of bytes still in the buffer/cache
I typed eject immediately … it hung for over a minute before returning the prompt — ie it was flushing the buffer before completing the eject.
The sync after the eject returned immediately … ie there was nothing left to flush.
Normally, if I typed sync immediately after the rsync finished, it would also take some time before returning.
Therefore either sync or eject will flush the buffer.
I did the whole thing again with umount instead of eject, and same result … umount waits until the buffer is flushed before returning.
If you copy any material to a removable drive ( with dd or rsync or cp or some other utility) , it is likely that the copy routine will return the user’s prompt, before the buffer/cache is fully flushed. Therefore you should use one of eject, umount or sync, and wait for it to return the prompt, before attempting to unplug the drive.
So @Rosika was OK using eject and what @daniel.m.tripp does with sync is also OK. What @Akito said about sync not being necessary is also right, provided one uses umount or eject before unplugging the drive.
Just a note on the difference
sync flushes the buffer(s) but leaves the drive mounted and present in Thunar or as an icon
umount flushes the buffer and then unmounts the drive filesystem, so it will not appear in a df table, but leaves the device present in Thunar and as an icon
eject flushes the buffer and then unmounts the filesystem so it will not appear in a df table, and then also removes the device so it will not appear in Thunar or as an icon. In some cases eject requires umount first.
What happens if you make a mistake and do a umount or eject while the copy process is still running? It depends who you are. For a normal user, it fails, for superuser it kills the copy process so it does not complete. So dont make that mistake!
So this answers the original questions 2 and 3.
You should use umount or eject before unplugging a drive after a write
You should use sync after a write , if you plan to read back material from the drive.
And in both cases you should wait for the eject or umount or eject to complete before proceeding.
Using Linux Mint ,I just plug-in or remove any USB flash stick without any other operation . In both cases I just hear a THUMB and that is it.
Sofar I never damaged the stick or crippled any file on it