NTFS

From Wikipedia, the free encyclopedia

NTFS is the standard file system of Windows NT, including its later versions Windows 2000, Windows XP, Windows Server 2003, Windows Server 2008, Windows Vista, and Windows 7.^[4]

NTFS supersedes the FAT file system as the preferred file system for Microsoft’s Windows operating systems. NTFS has several improvements over FAT and HPFS (High Performance File System) such as improved support for metadata and the use of advanced data structures to improve performance, reliability, and disk space utilization, plus additional extensions such as security access control lists (ACL) and file system journaling. The file system specification is a trade secret, although it can be licensed commercially from Microsoft through their Intellectual Property licensing program.

History

In the mid 1980s Microsoft and IBM formed a joint project to create the next generation graphical operating system. The result of the project was OS/2, but eventually Microsoft and IBM disagreed on many important issues and separated. OS/2 remained an IBM project. Microsoft started to work on Windows NT. The OS/2 filesystem HPFS contained several important new features. When Microsoft created their new operating system, they borrowed many of these concepts for NTFS.^[10] Probably as a result of this common ancestry, HPFS and NTFS share the same disk partition identification type code (07). Sharing an ID is unusual since there were dozens of available codes, and other major filesystems have their own code. FAT has more than nine (one each for FAT12, FAT16, FAT32, etc.). Algorithms which identify the filesystem in a partition type 07 must perform additional checks.

When it was first introduced, the acronym NTFS was understood to stand for New Technology File System. It was later abbreviated to NT File System and then to NTFS.^{[citation needed]}

Versions

NTFS has five released versions:

• v1.0 with NT 3.1,^{[citation needed]} released mid-1993
• v1.1 with NT 3.5,^{[citation needed]} released fall 1994
• v1.2 written by NT 3.51 (mid-1995) and NT 4 (mid-1996) (occasionally referred to as “NTFS 4.0″, because OS version is 4.0)
• v3.0 from Windows 2000 (“NTFS V5.0″)
• v3.1 from Windows XP (autumn 2001; “NTFS V5.1″), Windows Server 2003 (spring 2003; occasionally “NTFS V5.2″), Windows VistaWindows Server 2008 (mid-2005) (occasionally “NTFS V6.0″) and

V1.0 and V1.1 (and newer) are incompatible: that is, volumes written by NT 3.5x cannot be read by NT 3.1 until an update on the NT 3.5x CD is applied to NT 3.1, which also adds FAT long file name support.^[11] V1.2 supports compressed files, named streams, ACL-based security, etc.^[1] V3.0 added disk quotas, encryption, sparse files, reparse points, update sequence number (USN) journaling, the $Extend folder and its files, and reorganized security descriptors so that multiple files which use the same security setting can share the same descriptor.^[1] V3.1 expanded the Master File Table (MFT) entries with redundant MFT record number (useful for recovering damaged MFT files).

Windows Vista introduced Transactional NTFS, NTFS symbolic links, partition shrinking and self-healing functionality^[12] though these features owe more to additional functionality of the operating system than the file system itself.

Features

NTFS v3.0 includes several new features over its predecessors: disk usage quotas, sparse file support, reparse points, distributed link tracking, and file-level encryption, also known as the Encrypting File System (EFS).

Alternate data streams (ADS)

Alternate data streams allows files to be associated with more than one data stream. For example, a file such as text.txt can have an ADS with the name of text.txt:secret (of form filename:streamname) that can only be accessed by knowing the ADS name or by specialized directory browsing programs. Alternate streams are not detectable in the original file’s size but are lost when the original file (i.e. text.txt) is deleted with a RemoveFile or RemoveFileTransacted call (or a call that uses those calls), or when the file is copied or moved to a partition that doesn’t support ADS (e.g. a FAT partition, a floppy disk, or a network share). While ADS is a useful feature, it can also easily eat up hard disk space if unknown either through being forgotten or not being detected. It can also be used as a vector for malware to hide itself, although tools are available to scan for and remove such infections.

Quotas

Disk quotas were introduced in NTFS v3. They allow the administrator of a computer that runs a version of Windows that supports NTFS to set a threshold of disk space that users may utilize. It also allows administrators to keep track of how much disk space each user is using. An administrator may specify a certain level of disk space that a user may use before they receive a warning, and then deny access to the user once they hit their upper limit of space. Disk quotas do not take into account NTFS’s transparent file-compression, should this be enabled. Applications that query the amount of free space will also see the amount of free space left to the user who has a quota applied to them.

Sparse files

Sparse files are files which contain sparse data sets, data mostly filled with zeros. Many scientific applications can generate very large sparse data sets. Because of this, Microsoft has implemented support for efficient storage of sparse files by allowing an application to specify regions of empty (zero) data. An application that reads a sparse file reads it in the normal manner with the file system calculating what data should be returned based upon the file offset. As with compressed files, the actual size of sparse files are not taken into account when determining quota limits.^[13][14]

Reparse points

This feature was introduced in NTFS v3. These are used by associating a reparse tag in the user space attribute of a file or directory. When the object manager (see Windows NT line executive) parses a file system name lookup and encounters a reparse attribute, it knows to reparse the name lookup, passing the user controlled reparse data to every file system filter driver that is loaded into Windows 2000. Each filter driver examines the reparse data to see if it is associated with that reparse point, and if that filter driver determines a match then it intercepts the file system call and executes its special functionality. Reparse points are used to implement Volume Mount Points, Directory Junctions, Hierarchical Storage Management, Native Structured Storage, Single Instance Storage and Symbolic Links

Volume mount points

Similar to Unix mount points, where the root of another file system is attached to a directory. In NTFS, this allows additional file systems to be mounted without requiring a separate drive letter (like C: or D:) for each.

Directory junctions

Similar to volume mount points, however directory junctions reference other directories in the file system instead of other volumes. For instance, the directory C:\exampledir with a directory junction attribute that contains a link to D:\linkeddir will automatically refer to the directory D:\linkeddir when it is accessed by a user-mode application.^[1] This function is conceptually similar to symbolic links to directories in Unix except that the target in NTFS must always be another directory (typical Unix file systems allow the target of a symbolic link to be any type of file).

Hard links

Originally included to support the POSIX subsystem in Windows NT^[15], hard links are similar to directory junctions, but used for files instead of directories. Hard links can only be applied to files on the same volume since an additional filename record is added to the file’s MFT record. Short (8.3) filenames are also implemented as additional filename records that don’t have separate directory entries.

Hierarchical Storage Management (HSM)

Hierarchical Storage Management is a means of transferring files that are not used for some period of time to less expensive storage media. When the file is next accessed the reparse point on that file determines that it is needed and retrieves it from storage.

Native Structured Storage (NSS)

NSS was an ActiveX document storage technology that has since been discontinued by Microsoft. It allowed ActiveX Documents to be stored in the same multi-stream format that ActiveX uses internally. An NSS file system filter was loaded and used to process the multiple streams transparently to the application, and when the file was transferred to a non-NTFS formatted disk volume it would also transfer the multiple streams into a single stream.^[16]

Volume Shadow Copy

The Volume Shadow Copy Service (VSS) keeps historical versions of files and folders on NTFS volumes by copying old, newly-overwritten data to shadow copy (copy-on-write). The old file data is overlaid on the new when the user requests a revert to an earlier version. This also allows data backup programs to archive files currently in use by the file system. On heavily loaded systems, Microsoft recommends setting up a shadow copy volume on a separate disk.^[17]

File compression

NTFS can compress files using a variant of the LZ77 algorithm (also used in the popular ZIP file format).^[18] Although read-write access to compressed files is transparent, Microsoft recommends avoiding compression on server systems and/or network shares holding roaming profiles because it puts a considerable load on the processor.^[19]

Single-user systems with limited hard disk space will probably use NTFS compression successfully.^{[citation needed]} The slowest link in a computer is not the CPU but the speed of the hard drive, so NTFS compression allows the limited, slow storage space to be better used, in terms of both space and (often) speed.^[20] NTFS compression can also serve as a replacement for sparse files when a program (e.g. a download manager) is not able to create files without content as sparse files.

Single Instance Storage (SIS)

When there are several directories that have different, but similar, files, some of these files may have identical content. Single instance storage allows identical files to be merged to one file and create references to that merged file. SIS consists of a file system filter that manages copies, modification and merges to files; and a user space service (or groveler) that searches for files that are identical and need merging. SIS was mainly designed for remote installation servers as these may have multiple installation images that contain many identical files; SIS allows these to be consolidated but, unlike for example hard links, each file remains distinct; changes to one copy of a file will leave others unaltered. This is similar to copy-on-write, which is a technique by which memory copying is not really done until one copy is modified.^[21]

Encrypting File System (EFS)

EFS provides strong^[22] and user-transparent encryption of any file or folder on an NTFS volume. EFS works in conjunction with the EFS service, Microsoft’s CryptoAPI and the EFS File System Run-Time Library (FSRTL). EFS works by encrypting a file with a bulk symmetric key (also known as the File Encryption Key, or FEK), which is used because it takes a relatively smaller amount of time to encrypt and decrypt large amounts of data than if an asymmetric key cipher is used. The symmetric key that is used to encrypt the file is then encrypted with a public key that is associated with the user who encrypted the file, and this encrypted data is stored in an alternate data stream of the encrypted file. To decrypt the file, the file system uses the private key of the user to decrypt the symmetric key that is stored in the file header. It then uses the symmetric key to decrypt the file. Because this is done at the file system level, it is transparent to the user.^[23] Also, in case of a user losing access to their key, support for additional decryption keys has been built in to the EFS system, so that a recovery agent can still access the files if needed.

Symbolic links

Symbolic links (or soft links) were introduced in Windows Vista. Symbolic links are resolved on the client side. So when a symbolic link is shared, the target is subject to the access restrictions on the client, and not the server.

See also: Computer shortcut

Transactional NTFS

As of Windows Vista, applications can use Transactional NTFS to group changes to files together into a transaction. The transaction will guarantee that all changes happen, or none of them do, and it will guarantee that applications outside the transaction will not see the changes until the precise instant they’re committed.^[24]

USN Journal

The USN Journal is a system management feature that records changes to files and directories on the volume.

Interoperability

Details on the implementation’s internals are closed, which makes it difficult for third-party vendors to provide tools to handle NTFS.