Data and Black Holes

General Linux Question
1

I am using Xubuntu 22.04 Jammy Jellyfish.

In a recent newsletter 8 Feb 2023 Abhishek wrote about /dev/null as the Black Hole of Linux see
What is /dev/null in Linux?
and this got me thinking about data security issues. I suspect that sending data to /dev/null is like sending it to a black hole at least after the system is powered down and the data no longer exists in RAM.

What happens when data is deleted from a file manager or by command rm? I suspect that the inode is deleted but the file can be recovered by a file recovery app so this is more like Hawkins black hole where information is preserved. Presumably the “deleted” files would be over-written eventually as other files are created but could exist for some time. I handle a few files received from other people containing sensitive information so when I have finished with them I shred them so that the information is lost, I hope.

The sensitive documents I handle are often received as attachments to emails. What happens when these emails and attachments are deleted eg from Thunderbird? I suspect that these files are recoverable. It is difficult to access specific emails and attachments other than through the Thunderbird menues so they cannot easily be shredded. Is there a way of shredding these files?

The small amount of sensitive data I deal with is unlikely to attract the attention of a sophisticated hacker so as a careful user and the usual precautions it seems unlikely that the “deleted” files would be accessed via malware. However if the PC/laptop was stolen a hacker would have unlimited time to recover “deleted” files using readily available file recovery apps.

Are these scenarios eg prolonged access to the PC/laptop, and subsequent stealing of sensitive data realistic? If so what are the alternatives for ensuring safety of data? What should I recommend to a colleague handling the same sensitive data who is not very computer literate and who uses Win10?

1 Like
2

I think encrypting your disk is about the only workable solution. Then everything is protected automatically.
To have to chase individual files and scrub them is too much of a burden . Just protect everything.

It is possible to have encrypted email. My understanding is the two parties have to cooperate to achieve this. I dont know the details. Maybe someone else can elaborate.

2 Likes
3

Hi all, :wave:

I think what @nevj suggested makes a lot of sense.

Yes, quite right.

From personal experience I may say convincing the other party to setup encryption is the most difficult part. :slightly_frowning_face:

OpenPGP, which is used by thunderbird comes with its own key management tools and it´s really easy to set up.

You can create the key pair (public and private key) without difficulties and providing the public (!) key to the other party is done either by uploading it to a public keyserver or (better still:) by sending it via an attachment.

Once the other party involved does the same and you have his/her public (!) key imported it´s super-easy to exchange encrypred e-mails. :slightly_smiling_face:

Plus: if I´m not completely mistaken all attachments you send with the e-mail are also encrypted. :+1:

But:

I guess that´s a completely differnt case then. Any person with (physical) access to the laptop would be able to decrypt encrypted e-mails together with their attachments as the key pair is stored locally…

So I guess that´s not the answer @Clivegg12 is looking for. Sorry.

Apart from that: it´s a splendid idea to encrypt e-mails (if you can talk the other party into using it as well). :wink:

Many greetings
Rosika :slightly_smiling_face:

1 Like
4

Hi @Clivegg12 , :wave:

That would be difficult, I guess.

When sending an e-mail (with or without attachment) to my yahoo mail account it´s stored in:

/home/rosika/.thunderbird/[my_profile]/Mail/pop.mail.yahoo.com/

The respective file is inbox and it´s a “Mailbox file” according to thunar.

I just sent a test message to my yahoo account and put it in an already existing sub-folder called “Aldi Talk”.

Now the new message is located here:
/home/rosika/.thunderbird/[my_profile]/Mail/pop.mail.yahoo.com/Inbox.sbd/ .

As far as the sub-folder “Aldi Talk” is concerned there are two items present:


ll Aldi*
-rw------- 1 rosika rosika 2,2M Mär  4 15:09 'Aldi Talk'
-rw-rw-r-- 1 rosika rosika  46K Mär  4 15:09 'Aldi Talk.msf'

Those .msf files are just indexes, so it´s the Aldi Talk file itself which contains the new mail.
This file - accorfing to the file command - is:

file Aldi\ Talk 
Aldi Talk: Unicode text, UTF-8 text, with CRLF, LF line terminators

but trying to make any sense of its contents (and even modifying it) with the help of a text editor would be a futile attempt, I guess. It´s hard to impossible to make any sense of the mostly encrypted content. It´ll come across simply as gibberish… :frowning_face: .

Sorry I couldn´t be of some better help…

Many greetings
Rosika :slightly_smiling_face:

1 Like
5

I have ratted through old thunderbird files with an editor. You can find the text of an email message buried in lots of binary gibberish. I dont recommend it. Dont know about attachments.

I dont like the way thunderbird stores my email files. I would much prefer viewable text files. What happens if thunderbird disappears… how do I read old emails?

1 Like
6

Hi Neville, :wave:

Thanks for confirming it. That´s what I wanted to express in my previous post as well. :wink:

Well, let´s hope that won´t be happening any time soon. :wink:

It may a be a lot of work but it can be done I guess… :wink:

Just install the ImportExportTools NG add-on for thunderbird.
I use it for creating a dedicated compendium of certain sub-folders.

  • Mark all the e-mails you want to save (e.g. click on the first item and then on the last item with SHIFT pressed and held down; then all your e-mails are marked)
  • right-click: save selected items
  • choose HTML format
  • choose a folder to save them to # now you have all the e-mails you selected saved as html files.
  • with help of the cat command you can even concatenate them into a single html file
    which makes it very convenient for looking up certain search-words.
  • display the html file with any web-browser

further info regarding ImportExportTools NG:

Features:

Menus:

  • Tools (most import and export functions including profiles)
  • Folders (most import and export functions)
  • Search dialog (export search results)
  • Selected messages (export or copy to clipboard)

Export:

  • Individual folder(s), optionally including subfolders (mbox format)
  • All folder messages in the following formats:
  • EML, HTML, PDF, CSV or plaintext
  • Export as individual files or a single file
  • Optionally include attachments
  • Export indexes as plain text or CSV
  • Extensive file naming convention options
  • Export messages from search dialog

Import:

  • Mbox files (including structure)
  • EML & EMLX files
  • Individual or all directory files

Profiles:

  • Export complete profile or just the mail files
  • Import profile
  • Auto profile backup with schedule on shutdown

Homepage: GitHub - thunderbird/import-export-tools-ng: Import Export Tools that supports Thunderbird 68/78/91/102

Many greetings
Rosika :slightly_smiling_face:

7

Hi Rosika,
Tbank you, I needed that.
Those detailed instructions will help me get going. There are some old folders I need to keep.
Regards
Neville

1 Like
8

I wanted to play Age Of Empires II HD Edition (via SteamPlay) yesterday - and discovered to my severe dismay, that

  1. steam does NOT backup “scenario” files (maps) to the cloud - it ONLY backs up saved games
  2. steam backup (menu option in the client) is a piece of GARBAGE barely worth the effort - and a steam backup I created to include Age Of Empires II did NOT including “Age of Empires II”!

Doh! all that work - I spent hours and hours creating those maps / scenarios… Had they gone into an irretrievable black hole?

I looked across the room - there’s my old Linux Gaming desktop machine, Ubuntu 20.04, AMD Phenom II X6, 16 GB RAM, 2 GB NVidia GTX650… I know that will have what I want on the hard drive.

So I proceed to pull out the hard drive (an actual 3.5 " spinning thingie) only to discover, to remove the HDD I’d have to remove the CPU fan! Doh! What idiot built this PC (me)?

So - I have rig up a monitor, keyboard, put it on the network blah blah blah…

WOOHOO! There’s my aoe2 scenario files! And they weren’t even on the HDD - they were on the boot SSD (2.5" SATA SSD).

So - in future - I won’t be using Valve / Steams Backup and Restore utilitiy - I will be rsync’ing it to other storage!

1 Like
9

Hi Rosika and Neville many thanks for all the effort you have put in to the replies.

I should have mentioned that the sensitive emails I am concerned about may come from any of about 20 people so email encryption is a non starter.

Rosika’s explanation of the Thunderbird file structure prompted me to have a closer look at that. I sent myself an email with just ‘zzzzzzzz’ as subject and contents. went to the pop.gmail.com folder and did “grep -ir zzzzzzzz .” . Initially the email was found in the Inbox on lines with Subject and showing contents and also referenced in the .msf file. I then moved the message to “folderX” and grepped again. The message still appeared in the Inbox as Subject and contents and present in the .msf file but somewhere there must be a change to indicate that it does not display. The message also appears as expected in the “folderX” I then deleted it from folderX and grepped again when it appeared in Inbox, folderX and Trash. After deleting from Trash the message was still referenced in all three folders but of course the .msf files had been changed so that it does not display. Finally I compacted the emails when all references to the message were removed.
It seems that emails must be compacted to remove all copies of a message from the email folders but after compacting those copies are now in the state of Hawkins black hole ie the information will still remain until it is over-written.
It would be possible to extract the sensitive emails from the pop files but it may be more trouble than its worth. Much of the security could be achieved by compacting.
I will have another look at the import-export tools Rosika mentions, that may be a more secure route to follow.
With thanks Clive

1 Like
10

Hi all, :wave:

@nevj :

You´re welcome, Neville. :heart:

BTW:

An easy way to export the contents of an entire folder with the ImportExportTools NG add-on is this:

  • right-click on the respective (sub-) folder
  • choose “ImportExportTools NG”
  • choose “export all messages of the folder”
  • choose “HTML format”

That´ll give you a dedicated folder (following the path you chose).
Within it you get the items “messages” and “index.html”.

Although you can choose any of the exported mails one-by-one in the messages folder the “index.html” file may come in handy.
It´ll give you an overview of all the mails and you can open them from there as well just by clicking on any of the entries. :slightly_smiling_face:

Hope it helps.

@daniel.m.tripp :

Thanks, Dan, for your interesting account although you seem to have experienced some problems.
Sorry for that.

@Clivegg12 :

You´re welcome.

You seem to have done quite a lot of work youself.

Good to know.

O.K. I see that´s exactly what you want to avoid.
Perhaps someone else in the forum might be of better help.

Thanks a lot to all of you and many greetings
Rosika :slightly_smiling_face:

1 Like
11

Hi Rosika,
I am sorry for being so dumb,
Installed ImportExportTools NG’
I selected 2 emails for a test
It listed them, but it only offered archive or delete… there was nothing to click on to get ‘Save selected items’
If I clicked on Archive, it moved them to the Archive folder… I wanted it to save them to html?
What have I missed?

Regards
Neville

1 Like
12

Would you not be able to goto the account ( GMail or ???) that Thunderbird gets the Emails from ?

1 Like
13

Yes, if it is webmail… but I store messages in local folders and delete them on the webmail account.
Maybe that is the wrong way to do it. ?
It is the local folders that I am concerned about accessing, should thunderbird disappear.

What I would really like, is a mail reader that just stores its local folders as directories and text files in the Linux filesystem…, Then I could read them with any app I wanted.
What @Rosika has suggested (using ImportExportTools addon to export them as html), goes close to that… but I have to drive it manually. I would like it automatic

It really does annoy me when an app like thunderbird developes its own special filesystem, just to store its files. What is wrong with using the Linux filesystem? Ext and btrfs are enormously reliable and robust and will always be supported. Why use something else on top of the linux filesystem?
I am tempted to suggest that it is a ploy to try and lock users into their mailer, and that is against the spirit of FOSS.

1 Like
14

Hi again, :wave:

@nevj :

You´re certainly NOT dumb, Neville.
Let´s see if we can clear things up. Here´s a screenshot of my TB making use of the add-on in question.

Please bear in mind it´s in German but I will give you a translation.

kgw_3nana
kgw_3nana1041×701 138 KB

Here you see I right-clicked on the first e-mail “Bestätige Deineneue E-mail…” and a new context menu is visible now.

The first entry of this one is: “Ausgewählte Nachrichten speichern” (i.e. “save selected messages”).
Now I move the mouse-pointer over the first entry and another sub-menu is to be seen.
Here I choose the second entry: “Im HTML-Format” (i.e. "using html format").

The rest is simple. I just click on that one and choose a folder to save the e-mail to.

BTW: The last step throws two further messages at you which you may safely ignore. Just carry on and the e-mail is saved to the folder you selected in the process. :wink:

Hmm, I cannot imagine why that might be. :thinking:
Are you sure the ImportExportTools NG add-on is active?

Many greetings from Rosika :slightly_smiling_face:

15

Hi
@nevj

Evolution stores emails in separate text files but each file contains a lot of routing information and formatting information is also stored as plain text. You would need something like an email reader to be able to view the emails as received.
Data base files are in one of the Evolution directories so I suspect a database is used to keep track of which mail folder each email file is in etc. Clive

1 Like
16

Thanks Clive. I will have a look. It sounds rather like the original CLI mail program… it showed all the headers. I can live with that.

17

Hi Rosika,
Well, the answer is… I selected the email(s) by left click. That gives a list of selected emails and offers archive or delete only.
So I followed your beautiful instructions more carefully and selected an email with right click… and yes I get a popup of choices, one of which is Save to html file, which I did and it works.

I tried to send a screenshot but upload is not working?

I noticed there are other options, like save as a text file
I think right click on a whole folder, rather than individual emails, is likely to be more useful, because then the html can index it.
Thank you
Regards
Neville

Screenshot_2023-03-07_16-12-38
Screenshot_2023-03-07_16-12-381920×1080 298 KB

Got it. I dont know what the gliche was.
You can see your email in HTML onthe left in the browser

1 Like
18

@nevj :

Hi Neville, :wave:

Thanks for letting me know. I´m glad it works. :blush:

Yes, that´s what I was referring to in my post #10:

I guess that should be a convenient solution as it provides the index file as well.
You can get to any of the indexed e-mails from there by just clicking on them.

Many greetings from Rosika :slightly_smiling_face:

1 Like