NTFS is in some ways arguably still more advanced than ZFS: think reparse points... (2024)

psykotic on March 11, 2019 | next [–]

> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa....

Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons.

Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.

dataflow on March 12, 2019 | parent | next [–]

Some minor notes:

- For permissions WSL uses EAs; for capabilities it uses ADS

- I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.

setquk on March 11, 2019 | prev | next [–]

Not a lot of this matters in the real world. The real shame is the sh*tty small file performance due to MFT contention on NTFS.

otterley on March 11, 2019 | parent | next [–]

It absolutely matters: a fair amount of software, both enterprise and consumer, already uses these features. Malware detection, for example, makes heavy use of reparse points.

blattimwind on March 11, 2019 | root | parent | next [–]

Windows Update and Windows Installer use NTFS transactions.

setquk on March 11, 2019 | root | parent | next [–]

From Microsoft's documentation:

Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.

https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...

dataflow on March 11, 2019 | root | parent | next [–]

I'm pretty convinced that documentation is BS. I could be wrong I guess, but I honestly don't see them removing TxF anytime soon. They didn't exactly build their business on breaking backwards-compatibility of APIs.

setquk on March 11, 2019 | root | parent | next [–]

They've burned a lot of APIs in the last few years.

dataflow on March 11, 2019 | root | parent | next [–]

Examples?

setquk on March 11, 2019 | root | parent | next [–]

https://docs.microsoft.com/en-us/windows/desktop/apiindex/wi...

dataflow on March 11, 2019 | root | parent | next [–]

They're not removed though? They're 'deprecated' as in they don't want you to depend on them, but they're still very much there and usable.

cryptonector on March 11, 2019 | root | parent | prev | next [–]

ZFS snapshots work well enough for that.

The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.

otterley on March 11, 2019 | root | parent | prev | next [–]

Did that change recently? I haven't used Windows since XP, but back then, they used Volume Shadow Copies (VSS) for making backups used for reverting changes, not NTFS transactions.

tjalfi on March 12, 2019 | root | parent | next [–]

Windows Update switched to using NTFS transactions in Windows Vista.

setquk on March 11, 2019 | root | parent | prev | next [–]

Reparse points could be emulated at API level.

ygra on March 11, 2019 | parent | prev | next [–]

Is the MFT even documented and available via APIs on Windows? Could NTFS (at the expense of losing backwards compatibility in the file system (which already only extends to features that have been there before)) just change how the MFT works transparently for well-behaved applications (i.e. those that do not grovel around in undocumented data structures)? Admittedly, there may still be a number of applications that decide to »optimize« a system by digging around in parts that they have reverse engineered, and those might simply corrupt the file system when running on a newer NTFS version, but I guess that problem may have existed previously as well.

setquk on March 11, 2019 | root | parent | next [–]

MFT is an internal structure. They could fix it TBH but the issue is that this would require an NTFS filesystem revision and for those of us who suffered through NT4's broken ass filesystems might start complaining :)

chungy on March 12, 2019 | prev | next [–]

> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration"

ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.

otterley on March 12, 2019 | parent | next [–]

Reparse points and ADS? Do you have a reference handy? I’ve never seen such things or anything like them in ZFS.

chungy on March 12, 2019 | root | parent | next [–]

Reparse points are basically more primitive directory-only symlinks, aren't they? Even the symlinks introduced in Vista are better...

As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)

otterley on March 12, 2019 | root | parent | next [–]

> Reparse points are basically more primitive directory-only symlinks, aren't they?

Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power.

See https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.

chungy on March 12, 2019 | root | parent | next [–]

Thanks, I wasn't aware of that... terms seem to get interchanged in the Windows world often enough.

It doesn't sound like it is really an inherent property of the file system, but just controlled by a driver. I don't see why they couldn't be made to work on any other file system, such as FAT or ZFS.

ygra on March 12, 2019 | root | parent | prev | next [–]

Reparse points are directory entries that are special and interpreted by the file system driver (or some other higher-level component that's not the file system itself). Directory junctions (which you mean here, I guess) are reparse points, as are symlinks and a bunch of other things.

Gladdyu on March 11, 2019 | prev | next [–]

Though it might finally do away with the ever-persisting Windows error messages about "this file is already open" followed by a search through all of the processes that might have it open to terminate the offender.

otterley on March 11, 2019 | parent | next [–]

That error is orthogonal to the filesystem in use. Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

tracker1 on March 11, 2019 | root | parent | next [–]

 PS> $lockedFile="C:\Windows\System32\wshtcpip.dll" PS> Get-Process | foreach{$processVar = $_;$_.Modules | foreach{if($_.FileName -eq $lockedFile){$processVar.Name + " PID:" + $processVar.id}}}

jodrellblank on March 12, 2019 | root | parent | next [–]

 PS C:\> gps |? {$_.Modules.FileName -match "wshtcpip\.dll"} | select name, id

felixfbecker on March 11, 2019 | root | parent | prev | next [–]

That foreach looks like it should really be a where

tracker1 on March 11, 2019 | root | parent | next [–]

It was copied from stackoverflow... apologies, I just knew it was possible.

reitanqild on March 11, 2019 | root | parent | prev | next [–]

> Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

Obviously.

The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own.

I think that is the point GP tries to make.

Negitivefrags on March 11, 2019 | root | parent | next [–]

Open up "Resource Monitor" and go to the "Disk" tab. Under "Disk Activity" you can see all open file handles.

kevin_thibedeau on March 11, 2019 | root | parent | next [–]

Certainly not a sensible solution like having a dialog button that gives you all the relevant information immediately and empowers you to solve the problem without a goose chase.

coldtea on March 11, 2019 | root | parent | prev | next [–]

Which are thousands...

psykotic on March 12, 2019 | root | parent | next [–]

You need the Sysinternals utilities to make life tolerable: https://docs.microsoft.com/en-us/sysinternals/downloads/hand... from the command line or Ctrl-F in https://docs.microsoft.com/en-us/sysinternals/downloads/proc... if you prefer a GUI. They do a brute-force search over the open handles in all processes, but it's effective.

noisem4ker on March 13, 2019 | root | parent | next [–]

I prefer this utility: http://www.emptyloop.com/unlocker/

It lets you see which process locked a file, and optionally remove the lock.

kevingadd on March 11, 2019 | parent | prev | next [–]

The way filesystem locking works on Windows is an intentional decision and not an NTFS thing. It's there for a good reason even if it's inconvenient. The alternative has some real downsides. "Two CMD windows have the same CWD but are showing different folders" is not a user-friendly experience.

blibble on March 11, 2019 | root | parent | next [–]

I bet pretty much every Windows user would trade confusing command prompts in strange circ*mstances for not having to reboot for pretty much every single Windows Update (due to it being unable to update files that are in use)

muststopmyths on March 11, 2019 | root | parent | next [–]

It would be horrendous to even try to troubleshoot a system with a dozen different version of OS DLLs loaded because the system had not been rebooted for a dozen patches.

Or, imagine every copy of Word you have running is using a different set of binaries.

No, thank you.

kevin_thibedeau on March 11, 2019 | root | parent | next [–]

And yet Linux distros some how manage years of uptime with continuous updates.

muststopmyths on March 11, 2019 | parent | prev | next [–]

download sysinternals tools, one of which is "handle" for quickly finding open files.

Yeah, it should come with the OS, but that's Windows for you.

rkagerer on March 11, 2019 | parent | prev | next [–]

Send To > Unlocker

voltagex_ on March 11, 2019 | root | parent | next [–]

Hasn't worked since XP, has it? I'd love to see the source for that - I don't know how they were hooking the Copy dialog.

rkagerer on March 14, 2019 | root | parent | next [–]

http://www.emptyloop.com/unlocker/

I use it in Win 7; I think it works most of the time (not always).

usefulcat on March 12, 2019 | prev | next [–]

If a file is damaged by bit rot, NTFS will happily serve up garbage. So those are nice features, but viewed in that light they're little more than window dressing.

Demiurge on March 12, 2019 | parent | next [–]

You say that as if bit rot is the black plague of data storage. In the past 20 years I have encountered 'bit rot' with NTFS... not once. I have encountered bad blocks and replaced drives once in a while by monitoring SMART status, which probably prevented the rot. It is definitely great to have protection against it, but to call other features window dressing when others have used them more often than they thought of checksumming feature is not objectively fair.

usefulcat on March 12, 2019 | root | parent | next [–]

In the past 20 years I've lost multiple files to bit rot on NTFS. In my opinion, the primary purpose of a filesystem is to reliably store data. Checksumming is not particularly complex technically, does not impose significant run time costs, and allows the filesystem to at least detect when data has been damaged. Unless there's some other very compelling reason to not do checksumming, it's reasonable to consider it's omission to be a de facto bug.

Demiurge on March 12, 2019 | root | parent | next [–]

Does this mean ext4 is bugged too? We many filesystems with this bug, and we lived with this bug for a while. Anyway, I don't disagree on the technical cost/benefit analysis, but still, unfair.

kuwze on March 11, 2019 | prev [–]

Didn’t know any of that. Thanks!