Home » Questions » Computers [ Ask a new question ]

Bad NTFS performance

Bad NTFS performance

Why is it that NTFS performance is so lousy compared to, for example, Linux/ext3? Most often I see this when checking out (large) source trees from Subversion. Checkout takes around 10-15 minutes on NTFS, while corresponding checkout on Linux (on almost identical hardware) takes an order of magnitude faster (1 - 1.5 minutes).

Asked by: Guest | Views: 208
Total answers/comments: 2
Guest [Entry]

"NTFS has this thing called a Master File Table. It sounds really cool when you read about it.

You can see that ext3 performs alright up to about 95% disk use, while the existence of the MFT means that NTFS doesn't really want you to use more than 90% of your disk. But I'll assume that's not your problem, and that your problem is with the many operations on many small files.

One of the differences here is what happens when you create a small file. If a file is smaller than a block size, it is not written to it's own block but rather is stored in the MFT. This is nice if the file stays exactly the way it was when created. In practice though, it means that when svn touches a file to create it, then adds to that file, removes from it, or just modifies it by not enough to move it to it's own block, the operation is pretty slow. Also just reading lots of small files puts some stress on the MFT where they all reside, with multiples per block. Why would it do this? It's preemptively avoiding fragmentation and using more of the blocks more effectively, and in general that's a good thing.

In ext2 and 3 by contrast, file blocks for every file are stored next to where the directory metadata is for the directory they're in (when possible, if your disk is unfragmented and you have about 20% space free). This means that as svn is opening up directories, a number of blocks get cached basically for free in that 16mb cache on your drive, and then again in the kernel's cache. Those files might include the .svn file and the revision files for your last update. This is handy since those are likely some of the files svn is looking at next. NTFS doesn't get to do this, though large parts of the MFT should be cached in the system, they might not be the parts you will want next."
Guest [Entry]

Here's Microsoft's info on how NTFS works. It may be overkill for what you're looking for but studying it may shed some light on what scenarios NTFS has problems with.