Something that might count Against FAT32

One fact which In have written about, is that each File System essentially has a different way, of organizing the Blocks on an HD, or on an SSD, into Files, and of managing free storage space.

One fact which they all seem to have in common however, is that HD or SSD Blocks possess ‘Logical Block Addresses’, aka LBAs. What this means in simplified terms, is that every Block on an HD has a unique Block-Number, which can be used to point to it, very much the way in which Pointers do, to addresses in RAM. There can be some differences though, in whether the Physical Blocks correspond directly to Logical Blocks, mainly because a given HD platter might have a Physical Block Size of 512 Bytes, while the O/S might require that Logical Blocks have a size, compatible with the size of Pages, of Virtual Memory. Since Virtual Memory Pages are usually 4KB, this would seem to imply that 1 Logical Block always corresponds to 8 Physical Blocks, and that any file will take up space, corresponding to at least 1 Logical Block, regardless of where the logical end of the data is, inside its last Block.

One way in which a File System can allocate Blocks to a File, is just by chaining them, and this can make for an entertaining way to explain Computing. I.e., at the beginning of each allocated Block, there could be a Header, which among other things, states an LBA which points to a ‘Next Block’ in a chain, or which is otherwise Null to state that the current block is also the last one, belonging to a File.

The use of such header information that resides inside the Physical Blocks, that are supposed to contain data and not meta-data, has a major drawback though. It can be accommodated by a modern O/S with some effort, but again causes the Physical Blocks, not to correspond directly to Virtual Memory Pages, since some odd number of Bytes have to be subtracted from their size then, to arrive at the size of units of data and not meta-data, which then also does not correspond to a clean power of two.

I think that ‘FAT32′ uses such an approach.

Because a modern O/S has such features as Memory-Mapped Files and Virtual Memory, its performance would take a considerable hit, if it was forced to use the old ‘FAT32′ system – as its main store of system data.

What tends to happen with up-to-date File Systems, is that Logical Blocks that are mapped directly from HD Physical Blocks – Contain only Data, and contain no Meta-Data. ‘Unix System V’ was an early example of that; therefore ‘ext2′, ‘ext3′, ‘reiserfs’, and ‘ext4′ are examples of that as well. The way in which these File Systems keep track of Blocks as belonging to Files, is through one ‘iNode’ per File, plus an arbitrary number of Index Tables, the latter of which each contain one Logical Block of Meta-Data, consisting entirely of the LBAs, of data-blocks, that belong contiguously to any one File.

So there are entire pages of meta-data, stored on these higher-performing File Systems, and many of them. If that becomes corrupted, the effect is not the same, as when regular content-data becomes corrupted.

‘NTFS’ also possesses Index Tables, but I think replaces the use of ‘iNodes’ by Linux, with a ‘Master File Table’. LBAs are used as pointers throughout. Whether other people will think that a Master File Table is more secure than the use of many iNodes is, I do not know. I feel more secure with a system of iNodes.

The iNode or the MFT will state as part of their meta-data, the exact length of the File, in Bytes, so that the FS does not read past its end, even if the end of the content-data is midway through a Logical Block.

Dirk

 

I appreciate Plug-and-Play USB Keys.

I am used to an older time, when the stores did sell customers USB Drives / Keys, which were formatted with NTFS, and which, if we inserted them into our Linux PCs, would not mount. Today, the USB Keys we buy, can in some cases be plug-and-play, with modern Linux computers.

But the gap that once existed here, mainly existed between what a Linux computer could mount from the command-line, and what the desktop manager / HAL daemon was willing to mount, simply because the user plugged it in.

For example, I can perfectly-well format my USB Pen Drive to the ‘ext4‘ file system, but then the desktop manager would not accept that, because to do so would imply a security violation. It would effectively give any user ‘root‘, because any ext4 content could be stored as root, with ‘setiud, and be as the user provided. And yet, with the root password, it would still be possible to mount that…

There is a word of caution I should give fellow computer users though. Whichever computer you are on – be it Windows, OSX, Linux or other – you should still assume that if you mount a file system other than one native to your O/S, it will be mounted without journaling enabled. This means, that we will need to be extra careful, not to pull the key out, without first having done an explicit Safe Disconnect / Unmount.

It is the file system journal which will ensure, that only files are lost or damaged, which we were editing, and if so, atomically altered. Without this in place, we can end up with a USB Key which will only mount again explicitly in read-only mode, which also means that we will need to reformat again, before we can write new data to it. Hence, with no journaling, if the USB Key is pulled out while active, we could lose the whole content – although that is not usually what happens.

And so it could be said that Linux kernels as early as 2.4 had some ability to mount NTFS file systems, it would have been premature to say that those would be fully compatible with the USB Key, as provided, just because the desktop managers did not mount it. Even though they could already be mounted from the command-line.

I guess that the people who configure Linux these days, have figured out, that for their desktop manager to auto-mount an NTFS volume, does not present a security risk.

There does exist a type of NTFS driver under Linux, which will do an NTFS mount, more the way Windows does NTFS. It is called ‘ntfs-3g‘. But the only real purpose of that is to allow the owners of dual-boot laptops, to mount their Windows partitions from within Linux, which remains the only easy way to exchange data between Linux and Windows sessions. ntfs-3g was not meant for USB Keys, nor for automatic service from HAL.


Speaking of which: I just needed to trash a USB Key. I was able to mount it under Linux, and then to save a folder of data to it, which seemed to join the already-present folders in ‘Dolphin‘. But after that, whenever I removed the USB Key and remounted it, all the changes I had made were just gone. It seemed as though they had been cached, and with the cache cleared, the actual mass-storage device did not remember anything. This was accompanied by some very strange error messages, whenever I tried to repair the old key with software tools.

And so at some point I had to acknowledge that either the H/W had to have become defective, or that I had done so much reformatting, that I could reformat the little thing no more. Scientifically, the second option makes less sense, because no matter how many times one reformats the key back to FAT32 finally, its configuration should go back to one that at least works, even if it can only handle up to 40GB total. Mine would not recover into this state.

And it should also have been possible to format it with ‘etx2‘ on one specific Linux computer, to store data on that partition, to remove, reinsert and then remount the key on the same computer, and to retrieve the data. This experiment had also failed, at the point where the key had been remounted but failed to show the new data.

So now I have a brand-new USB key with a capacity for 128GB, which is plug-and-play compatible with my Linux boxes, and which remembers everything I stored on it (Insert Evil Scientist Laugh Here.)

I did not even bother to open it with ‘gparted‘ first, just to satisfy my curiosity. I do not care what FS they used. I just tried plugging it in, and it worked. And my two main Linux machines are on exactly the same build.

I had bought the old USB Key, before this present computer – ‘Phoenix’ – was installed, when its hardware was host to a system I had named ‘Thunderbox’. And Thunderbox was not able, to mount that older USB Key directly.

However, that USB Key was useful to me, in migrating some of my data, around the time that I was reinstalling ‘Aphrodite’ as ‘Venus’, and ‘Thunderbox’ as ‘Phoenix’. The key was formatted with FAT32 and held its data at the time. But what I really wanted today was a full-capacity USB Key, not limited to 40GB.

I would bet, that if I had re-partitioned it with FAT32 using my Windows PC, it would have played its old game with me: Of seeming to store data for a longer time, but then just forgetting it again.

What I needed was a new USB Key, which I now have.

Dirk

(Edit 06/06/2016 : ) I just found out, that what Sony did, was just to format this new USB Key with FAT32.

A part of that which I do not understand, and which contributed to my initial confusion, was my memory to the effect that, FAT32 file systems were once not allowed to be bigger than 40GB. But this is a 128GB stick. I do not understand this.