There have been a number of issues raised recently regarding USB flash drives.
Could someone give definitive answers to the following
is it safe to unplug a drive while the computer is powered on?
should a drive be unmounted before removal?
should sync be used before unmounting or power off?
should sync be used after writing to a usb drive?
what if the computer is rebooted rather than powered off?
what if one uses halt or shutdown
what if the computer is in suspend or hibernate?
what if a drive is used on Win as well as Linux?
I must admit I have always been super conservative and never remove a drive until the computer is completely powered offā¦ But they are supposed to be plug-n-play arent they?
I must admit I may not be knwledgeable enough to answer all of your questions.
However thatĀ“s the way I do it:
If it for any reason might be necessary to disconnect a USB flash drive I donĀ“t think you must power down the system beforehand.
Using thunar as my daily file manager thereĀ“s the option āejectā when right-clicking on the
left-hand side (ādevicesā).
So I just eject the device and then - I guess - itĀ“s safe to remove it.
Before shutting down my system I do the same. No idea if thatĀ“s totally neccessary; I just do it anyway.
Hmm, I think that question should be reserved for someone elso to answer. IĀ“ve been on Linux/Lubuntu for about six years now ā¦ exclusively.
As far as reboots are concerned I never bother with ejecting or unmounting them first.
Suspend and hibernate I never use.
So hopefully some other folks may be able to shed some light on the matter.
No need. Itās usually just about being paranoid.
No need. Itās usually just about being paranoid.
Depends if ārebootā is āenhancedā to not make it a real reboot anymore, as it is the default in Windows.
That turns off the computer.
Suspend and Hibernate need to be set up properly in Linux, to work properly. Otherwise, these modes may cause issues.
Not a great idea, but doable. Itās just that you would need to choose the lowest common denominator between the two, which is some FAT32 or NTFS file system, which in turn means you cannot do anything on Linux with files on these file systems, except read, write & copy/move them.
@Akito
I disagree with your No 4 answer
When I dd a large file to a usb drive at the command line, the dd returns a message saying āxā bytes read and āyā bytes written, and āxā is always much larger than āyā, so it has a huge number of bytes still in a memory bufferā¦ and it returns control to the user
So I issue a sync
and then it hangs for ages while it flushes the buffer, before returning a prompt.
Not sure what would happen if I tried an unmount instead of a `syncāā¦ never been game enought to try it
To me, dd should not be returning control to the user, before the buffer is flushed?
More questions:
Are memory cards the same?
Is @Rosika 's use of eject the same as unmount?
And thank you,
it makes more sense to me now, and hopefully to others
Neville
That Linux is badly designed then. It shouldnāt be ever necessary to sync manually. Nowadays, itās usually a sign that somethingās wrong, if itās truly necessary. Making the user watch a memory buffer is a UX nightmare.
Indeed. This should be default behaviour. The problem with dd is, itās a truly ancient tool. I wouldnāt recommend using it. If someone really wants a low-level dd-like tool, he should use ddrescue instead. Itās like dd but it actually is not a pain in the ass to use, as dd is.
Hi Rosika,
I tested eject in Thunar.
You are right, it unmounts the USB drive.
So unplugging it should be safeā¦ except I am not sure what would happen if you had written to the USB drive and it had not finished flushing the buffer when you ejected. I would hope eject would sync the buffer before unmounting. ( see discussion with @Akito about dd and sync)
Regards
Neville
Overkillā¦ if Iāve plonked data on there - I run a sync (especiall āsudo syncā if Iād been root doing a āddā or something), sometimes I eject (CLI or nautilus)ā¦ Mostly I donātā¦ So long as sync finished Iām happyā¦
Iāve only ever had dramas with removable media on Windowsā¦ I donāt run or use Windows anywhere these daysā¦ Hope to never go back there againā¦
I have heaps of Fat32 USB drives around - I plonk mp3 files on them for when Iām driving the carā¦ Iām slowly replacing my MP3 collection with FLAC, but the USB music player in the car (10 year old Toyota) wonāt play anything but mp3, or see anything but Fat32ā¦ Never had any problems there either - I strongly suspect thereās a Linux system behind the ugly UI on the Toyota music screen (thatās all it does - plays music, or interfaces with a phone - but donāt get me started on that - the Bluetooth on it is so shonky it can take 15 minutes to pair! 10 year old LINUX was GARBAGE for Bluetooth!).
Anyway - I insert and remove USB so often a full power cycle would be a huge PITA - also - considering I barely EVER shutdown or reboot my main desktop, I donāt plan on changing thatā¦
Well it is - it inherited this from UNIXā¦ I can remember running sync on Solaris all the time, not necessarily after dd, but I just got use to doing itā¦ Remember āddā is an old UNIX toolā¦ I used to use dd to create TAR files from raw TAR (TApe aRchive) images off floppy disksā¦ more reliable than direct read tar from the deviceā¦
I was not advocating everybody do shutdowns to remove a Usb drive. Its just easy for me because I always shutown when finished for the day. Your usage patterns are different.
I agree sync is necessaryā¦ because Unix utilities like dd or rsync or even cp, return control to the user, without telling you if or when the buffer is flushed.
That behaviour is fine for an HD, but not for removable devices.
This is something Linux/Unix need to fix.
IĀ“ve never tried this before but I may tell you the following:
When I change into the USB-stick-directory in thunar and open a terminal with āopen terminal hereā then the teminal-related process is connected to the (mounted) USB-stick, right?
Because the terminal-command pwd shows me e.g.
If I donĀ“t close the terminal (and havenĀ“t changed the directory) and then try to eject the USB-drive from within thunar it wonĀ“t let me by default.
A popup message appears saying something like āthereĀ“s still data being written to the ā¦ driveā.
I guess that message would also appear if any other process like dd was still making use of the drive.
ā¦ at least thatĀ“s my experience with thunar. Hope it helps.
Hi Rosika,
Yes, if you look at the umount or eject man pages, it says that umount or eject will not proceed if the device is ābusyā, which it clearly was in your case because your terminal was āinā the devices directory.
What I would like to know is what does ābusyā mean. In particular does it cover the case where you were writing some files to the usb drive, the write process terminated , but the buffers were still being flushed?
I guess that message would also appear if any other process like `dd` was still making use of the drive.
Yes, if dd were still running, that message would appear.
But the issue is that dd stops running, and returns control to the user, before the the write process is completed. It relies on the system sync to complete the emptying of the buffers.
We shall find the answer, but it is not clear in the man pages
Regards
Neville
What you see in action is the file buffer. First, content is copied to the file buffer. When the buffer is full, copying continues to the file buffer, while on the other end, content is physically being flushed to the disk.
Once copying is āfinishedā for the file manager, the buffer is still being flushed in the background until complete. This is how Linux works, and it will work the same with nautilus Files as with any other file manager on linux.
That you have gotten corrupt files is not the fault of Canonical. It is your fault. To correctly unmount a volume, you need to click the āejectā button in the file manager, or right-click and select āSafely unmount diskā.
If writing is still ongoing, you will get a notification that you should not yet unplug the drive. Once it is safe to eject the drive, you also will get a notification.
So await the notification before removing the drive. No software or instrumentation ever is 100% foolproof against unproper use.
ā¦
As I indicated in my answer, you do not need to guess. You just need to unmount properly using software: then the system informs you. Also on a server a device must be properly unmounted. If using the terminal, the system will tell you it cannot if writing still is ongoing.
Hi Rosika,
No,no. Nothing is obvious. We need the facts stated
In addition, Ubuntu will not tell you that itās safe to remove the device before the cache is flushed.
Well that is the bad news that I feared
If that is correct, then eject or umount will not protect the user from unmounting before the buffer is flushed
There is only one way to be sureā¦ I will get a spare usb drive, write something huge like an iso file to it, then when dd returns control to the user, immediately try to eject or umount it. We shall see
The point behind āunmountingā or āsafely removingā is that you have to make sure all the buffered data has been flushed ( or synced with the device) .
So if you just poweroff the device via their operating systems (i.e soft poweroff ) , it would be fine because all operating systems make sure all I/O operations are done before actually shutting down the device.
But if you mean a hard poweroff (e.g plugging out a HDD ) , it might cause data corruption or even in the worst case , filesystem corruption if you do this when a change in the partition table is being made
If youāre worried about data loss, you could create a new blank file , write a single character into it, save and then immediately press eject.
After remounting, if the file is there as well as the character you typed, then it is syncing properly before unmounting.
A single character would definitely stay in the buffer so thatās a good test. The eject will also fail if the drive is being used to avoid corruption.
In terminal, cd to your removable drive, type while true; do ls; done and then try to eject the disk. It should say itās busy.
Well I did the above experiment, except used rsync instead of dd.
Here is what happened
# rsync -aAXvH --exclude=Downloads/li* ./* /mnt/TestData
sending incremental file list
Downloads/
Downloads/tsetup.3.7.3.tar.xz
Downloads/waterfox-G4.1.1.1.en-US.linux-x86_64.tar.bz2
Downloads/waterfox-G4.1.2.en-US.linux-x86_64.tar.bz2
sent 207,566,275 bytes received 79 bytes 59,304,672.57 bytes/sec
total size is 229,587,244 speedup is 1.11
# eject /dev/sdb1
...... this ran for about a minute before returning the prompt ...
# sync
..... this returned the prompt immediately......
#
Look at the difference between sent and received at the end of the rsyncā¦ there is a lot of bytes still in the buffer/cache
I typed eject immediately ā¦ it hung for over a minute before returning the prompt ā ie it was flushing the buffer before completing the eject.
The sync after the eject returned immediately ā¦ ie there was nothing left to flush.
Normally, if I typed sync immediately after the rsync finished, it would also take some time before returning.
Therefore either sync or eject will flush the buffer.
I did the whole thing again with umount instead of eject, and same result ā¦ umount waits until the buffer is flushed before returning.
Conclusion:
If you copy any material to a removable drive ( with dd or rsync or cp or some other utility) , it is likely that the copy routine will return the userās prompt, before the buffer/cache is fully flushed. Therefore you should use one of eject, umount or sync, and wait for it to return the prompt, before attempting to unplug the drive.
So @Rosika was OK using eject and what @daniel.m.tripp does with sync is also OK. What @Akito said about sync not being necessary is also right, provided one uses umount or eject before unplugging the drive.
Just a note on the difference
sync flushes the buffer(s) but leaves the drive mounted and present in Thunar or as an icon
umount flushes the buffer and then unmounts the drive filesystem, so it will not appear in a df table, but leaves the device present in Thunar and as an icon
eject flushes the buffer and then unmounts the filesystem so it will not appear in a df table, and then also removes the device so it will not appear in Thunar or as an icon. In some cases eject requires umount first.
What happens if you make a mistake and do a umount or eject while the copy process is still running? It depends who you are. For a normal user, it fails, for superuser it kills the copy process so it does not complete. So dont make that mistake!
So this answers the original questions 2 and 3.
You should use umount or eject before unplugging a drive after a write
You should use sync after a write , if you plan to read back material from the drive.
And in both cases you should wait for the eject or umount or eject to complete before proceeding.
thank you so much for putting a lot of time and effort into solving the problem ā¦ and above all, letting us know what you came up with. ThatĀ“s really great.
IĀ“m sure there are some people who may benefit from your findings. At least thatĀ“s certainly true for me.
Thanks for confirming that.
Yes, thatĀ“s really great. IĀ“ll make a note of that in my how-to-folder .
Thanks once more for this great work of yours.
Many greetings from Rosika
Using Linux Mint ,I just plug-in or remove any USB flash stick without any other operation . In both cases I just hear a THUMB and that is it.
Sofar I never damaged the stick or crippled any file on it
