TL;DR<\/b><\/p>\n
If your app scans thousands of files, the way you fetch file attributes can quietly destroy performance. Standard methods like\u00a0<\/span> The real win? Combine faster APIs with smart strategies like caching, and your file-heavy workflows go from sluggish to lightning-fast without changing core logic.\u00a0<\/span><\/p>\n Imagine you’re developing a tool that needs to scan for file changes across thousands of project files. Retrieving file attributes efficiently becomes critical for such scenarios. In this article, I’ll demonstrate a technique to get file attributes that can achieve a surprising speedup of over 50+ times compared to standard Windows methods.<\/p>\n Let’s dive in and explore how we can achieve this.<\/p>\n This is a blog post made in collaboration with Bartlomiej Filipek from C++ stories. You can visit his blog here<\/a>.<\/em><\/strong><\/p>\n The inspiration for this article came from a recent update for Visual Assist – a tool that heavily improves Visual Studio experience and productivity for C# and C++ developers.<\/p>\n In one of their blog post, they shared:<\/p>\n The initial parse is 10..15x faster!<\/p><\/blockquote>\n What\u2019s New in Visual Assist 2024\u2014Featuring lightning fast parser performance [Webinar] – Tomato Soup<\/a><\/p>\n After watching the webinar, I noticed some details about efficiently getting file attributes and I decided to give it a try on my machine. In other words I tried to recreate their results.<\/p>\n Disclaimer<\/strong>: Idera, the company behind Visual Assist, helped me write this post and sponsored it.<\/p>\n On Windows, there are at least a few options to check for a file change:<\/p>\n Below, you can see some primary usage of each approach:<\/p>\n Additionally you can also pass Assuming you have a file open, which is a directory, you can iterate over its children in the following way:<\/p>\n Introduced in C++17, the To evaluate the performance of different file attribute retrieval methods, I developed a small benchmark. This application measures the time taken by each method to retrieve file attributes for N number of files in a specified directory.<\/p>\n Here’s a rough overview of the code:<\/p>\n The Each retrieval technique will have to go over a directory and build a vector of And the last one, the most portable technique:<\/p>\n The To measure the performance of each file attribute retrieval method, I executed benchmarks on a directory containing 1000, 2000 or 5000 random text files. The tests were performed on a laptop equipped with an Intel i7 4720HQ CPU and an SSD. I measured the time taken by each method and compared the results to determine the fastest approach.<\/p>\n Each test run consisted of two executions: the first with uncached file attributes and the second likely benefiting from system-level caching.<\/p>\n The speedup factor is the factor of the current result compared to the slowest technique in a given run.<\/p>\n 1000 files:<\/p>\n Directory with 2000 files:<\/p>\n Directory with 5000 random, small text files:<\/p>\n The results consistently showed that For the directory with 2000 files, Getting file attributes is only part of the story, and while important, they may contribute to only a small portion of the overall performance for the whole project. The Visual Assist team, who contributed to this article, improved their initial parse time performance by avoiding The main item being optimised was the initial parse, where VA builds a symbol database when a project is opened for the first time. This involves parsing all code and all headers. They decided that it’s a reasonable assumption that headers won’t change while a project is being loaded, and so the file access is cached during the initial parse, avoiding the filesystem entirely. (Changes after a project has been parsed the first time are, of course, still caught.) The combination of switching to a much faster method for checking filetimes and then avoiding file IO completely contributed to the up-to-15-times-faster performance improvement they saw in version 2024.1 at the beginning of this year.<\/p>\n Read further details on their blog Visual Assist 2024.1 release post – January 2024<\/a> and Catching up with VA: Our most recent performance updates – Tomato Soup<\/a>.<\/p>\n In the text, we went through a benchmark that compares several techniques for fetching file attributes. In short, it’s best to gather attributes at the same time as you iterate through the directory – using The benchmark also showed one feature: while C++17 and its Share your comments below. And if you’re using C++, you can also download and try Visual Assist yourself for 30 days for free.<\/p>\nGetFileAttributesEx<\/code>\u00a0or even <\/span>std::filesystem<\/span>\u00a0a<\/span>re convenient but painfully slow at scale. By switching to faster Windows APIs like FindFirstFileEx<\/code> or <\/span>GetFileInformationByHandleEx\u00a0<\/span>, you can retrieve attributes while iterating directories, cutting expensive system calls and achieving speedups of 50x or more.<\/span><\/p>\nThe inspiration<\/h2>\n
Understanding File Attribute Retrieval Methods on Windows<\/h2>\n
\n
FindFirstFile[EX]<\/code> – with Basic, Standard and LargeFetch options<\/li>\nGetFileAttributesEx<\/code><\/li>\nstd::filesystem<\/code><\/li>\nGetFileInformationByHandleEx<\/code><\/li>\n<\/ul>\nFindFirstFileEx<\/code><\/h3>\nFindFirstFileEx<\/code> is a Windows API function that allows for efficient searching of directories. It retrieves information about files that match a specified file name pattern. The function can be used with different information levels, such as FindExInfoBasic<\/code> and FindExInfoStandard<\/code>, to control the amount of file information fetched.<\/p>\nWIN32_FIND_DATA findFileData;\r\nHANDLE hFind = FindFirstFileEx((directory + \"\\\\*\").c_str(), FindExInfoBasic, &findFileData, FindExSearchNameMatch, NULL, 0);\r\n\r\nif (hFind != INVALID_HANDLE_VALUE) {\r\n do {\r\n \/\/ Process file information\r\n } while (FindNextFile(hFind, &findFileData) != 0);\r\n FindClose(hFind);\r\n}\r\n<\/code><\/pre>\nFIND_FIRST_EX_LARGE_FETCH<\/code> as the additional flag to indicate that the function should use a larger buffer which might bring some extra performance.<\/p>\nGetFileAttributesEx<\/code><\/h3>\nGetFileAttributesEx<\/code> is another Windows API function that retrieves file attributes for a specified file or directory. Unlike FindFirstFileEx<\/code>, which is used for searching and listing files, GetFileAttributesEx<\/code> is typically used for retrieving attributes of a single file or directory.<\/p>\nWIN32_FILE_ATTRIBUTE_DATA fileAttributeData;\r\nif (GetFileAttributesEx((directory + \"\\\\\" + fileName).c_str(), GetFileExInfoStandard, &fileAttributeData)) {\r\n \/\/ Process file attributes\r\n}\r\n<\/code><\/pre>\nGetFileInformationByHandleEx<\/code><\/h3>\nGetFileInformationByHandleEx<\/code> is a low level routine that might be tricky to use, but gives us more control over the iteration. The main idea is to get a lerge buffer of data and read it on the application side, rather than rely on sometimes costly kernel\/system calls.<\/p>\nwhile (true) {\r\n if (!GetFileInformationByHandleEx(\r\n hDir,\r\n FileFullDirectoryInfo,\r\n pInfo,\r\n sizeof(buffer))) {\r\n DWORD error = GetLastError();\r\n if (error == ERROR_NO_MORE_FILES) {\r\n break;\r\n }\r\n else {\r\n std::wcerr << L\"GetFileInformationByHandleEx failed (\" << error << L\")\\n\";\r\n break;\r\n }\r\n }\r\n\r\n do {\r\n if (!(pInfo->FileAttributes & FILE_ATTRIBUTE_DIRECTORY)) {\r\n FileInfo fileInfo;\r\n fileInfo.fileName = std::wstring(pInfo->FileName, pInfo->FileNameLength \/ sizeof(WCHAR));\r\n FILETIME ft{};\r\n ft.dwLowDateTime = pInfo->LastWriteTime.LowPart;\r\n ft.dwHighDateTime = pInfo->LastWriteTime.HighPart;\r\n fileInfo.lastWriteTime = ft;\r\n files.push_back(fileInfo);\r\n }\r\n pInfo = reinterpret_cast<FILE_FULL_DIR_INFO*>(\r\n reinterpret_cast<BYTE*>(pInfo) + pInfo->NextEntryOffset);\r\n } while (pInfo->NextEntryOffset != 0);\r\n}\r\n<\/code><\/pre>\nstd::filesystem<\/code><\/h3>\nstd::filesystem<\/code> library provides a modern and portable way to interact with the file system. It includes functions for file attribute retrieval, directory iteration, and other common file system operations.<\/p>\nfor (const auto& entry : fs::directory_iterator(directory)) {\r\n if (entry.is_regular_file()) {\r\n \/\/ Process file attributes\r\n auto ftime = fs:last_write_time(entry);\r\n ...\r\n }\r\n}\r\n<\/code><\/pre>\nThe Benchmark<\/h2>\n
FileInfo<\/code> struct stores the file name and last write time.<\/p>\nstruct FileInfo {\r\n std::wstring fileName;\r\n std::variant<FILETIME, std::filesystem::file_time_type> lastWriteTime;\r\n};\r\n<\/code><\/pre>\nFileInfo<\/code> objects.<\/p>\nBenchmarkFindFirstFileEx<\/h4>\n
void BenchmarkFindFirstFileEx(const std::string& directory, \t\r\n std::vector<FileInfo>& files, \r\n FINDEX_INFO_LEVELS infoLevel) \r\n{\r\n WIN32_FIND_DATA findFileData;\r\n HANDLE hFind = FindFirstFileEx((directory + \"\\\\*\").c_str(),\r\n infoLevel, \r\n &findFileData, \r\n FindExSearchNameMatch, NULL, 0);\r\n\r\n if (hFind == INVALID_HANDLE_VALUE) {\r\n std::cerr << \"FindFirstFileEx failed (\" \r\n << GetLastError() << \")\\n\";\r\n return;\r\n }\r\n\r\n do {\r\n if (!(findFileData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)) {\r\n FileInfo fileInfo;\r\n fileInfo.fileName = findFileData.cFileName;\r\n fileInfo.lastWriteTime = findFileData.ftLastWriteTime;\r\n files.push_back(fileInfo);\r\n }\r\n } while (FindNextFile(hFind, &findFileData) != 0);\r\n\r\n FindClose(hFind);\r\n}\r\n<\/code><\/pre>\nBenchmarkGetFileAttributesEx<\/h4>\n
void BenchmarkGetFileAttributesEx(const std::string& directory,\r\n std::vector<FileInfo>& files) \r\n{\r\n WIN32_FIND_DATA findFileData;\r\n HANDLE hFind = FindFirstFile((directory + \"\\\\*\").c_str(),\r\n &findFileData);\r\n\r\n if (hFind == INVALID_HANDLE_VALUE) {\r\n std::cerr << \"FindFirstFile failed (\" \r\n << GetLastError() << \")\\n\";\r\n return;\r\n }\r\n\r\n do {\r\n if (!(findFileData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)) {\r\n WIN32_FILE_ATTRIBUTE_DATA fileAttributeData;\r\n if (GetFileAttributesEx((directory + \"\\\\\" + findFileData.cFileName).c_str(), GetFileExInfoStandard, &fileAttributeData)) {\r\n FileInfo fileInfo;\r\n fileInfo.fileName = findFileData.cFileName;\r\n fileInfo.lastWriteTime = fileAttributeData.ftLastWriteTime;\r\n files.push_back(fileInfo);\r\n }\r\n }\r\n } while (FindNextFile(hFind, &findFileData) != 0);\r\n\r\n FindClose(hFind);\r\n}\r\n<\/code><\/pre>\nBenchmarkStdFilesystem<\/h4>\n
void BenchmarkStdFilesystem(const std::string& directory, \r\n std::vector<FileInfo>& files) \r\n{\r\n for (const auto& entry : std::filesystem::directory_iterator(directory)) {\r\n if (entry.is_regular_file()) {\r\n FileInfo fileInfo;\r\n fileInfo.fileName = entry.path().filename().string();\r\n FILETIME ft{};\r\n ft.dwLowDateTime = pInfo->LastWriteTime.LowPart;\r\n ft.dwHighDateTime = pInfo->LastWriteTime.HighPart;\r\n fileInfo.lastWriteTime = ft;\r\n files.push_back(fileInfo);\r\n }\r\n }\r\n}\r\n<\/code><\/pre>\nBenchmarkGetFileInformationByHandleEx<\/h4>\n
void BenchmarkGetFileInformationByHandleEx(const std::wstring& directory, std::vector<FileInfo>& files) {\r\n HANDLE hDir = CreateFileW(\r\n directory.c_str(),\r\n GENERIC_READ,\r\n FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,\r\n NULL,\r\n OPEN_EXISTING,\r\n FILE_FLAG_BACKUP_SEMANTICS,\r\n NULL\r\n );\r\n\r\n if (hDir == INVALID_HANDLE_VALUE) {\r\n std::wcerr << L\"CreateFile failed (\" << GetLastError() << L\")\\n\";\r\n return;\r\n }\r\n\r\n constexpr DWORD BufferSize = 64 * 1024;\r\n uint8_t buffer[BufferSize];\r\n FILE_FULL_DIR_INFO* pInfo = reinterpret_cast<FILE_FULL_DIR_INFO*>(buffer);\r\n\r\n while (true) {\r\n if (!GetFileInformationByHandleEx(\r\n hDir,\r\n FileFullDirectoryInfo,\r\n pInfo,\r\n sizeof(buffer))) {\r\n DWORD error = GetLastError();\r\n if (error == ERROR_NO_MORE_FILES) {\r\n break;\r\n }\r\n else {\r\n std::wcerr << L\"GetFileInformationByHandleEx failed (\" << error << L\")\\n\";\r\n break;\r\n }\r\n }\r\n\r\n do {\r\n if (!(pInfo->FileAttributes & FILE_ATTRIBUTE_DIRECTORY)) {\r\n FileInfo fileInfo;\r\n fileInfo.fileName = std::wstring(pInfo->FileName, pInfo->FileNameLength \/ sizeof(WCHAR));\r\n FILETIME ft{};\r\n ft.dwLowDateTime = pInfo->LastWriteTime.LowPart;\r\n ft.dwHighDateTime = pInfo->LastWriteTime.HighPart;\r\n fileInfo.lastWriteTime = ft;\r\n files.push_back(fileInfo);\r\n }\r\n pInfo = reinterpret_cast<FILE_FULL_DIR_INFO*>(\r\n reinterpret_cast<BYTE*>(pInfo) + pInfo->NextEntryOffset);\r\n } while (pInfo->NextEntryOffset != 0);\r\n }\r\n\r\n CloseHandle(hDir);\r\n}\r\n<\/code><\/pre>\nThe Main Function<\/h3>\n
main<\/code> function sets up the benchmarking environment, runs the benchmarks, and prints the results.<\/p>\nstd::wstring directory = argv[1];\r\nconst auto arg2 = argc > 2 ? std::wstring_view(argv[2]) : std::wstring_view{};\r\n\r\nstd::vector<std::pair<std::wstring, std::function<void(std::vector<FileInfo>&)>>> benchmarks = {\r\n {L\"FindFirstFileEx (Basic)\", [&](std::vector<FileInfo>& files) {\r\n BenchmarkFindFirstFileEx(directory, files, FindExInfoBasic, 0);\r\n }},\r\n {L\"FindFirstFileEx (Standard)\", [&](std::vector<FileInfo>& files) {\r\n BenchmarkFindFirstFileEx(directory, files, FindExInfoStandard, 0);\r\n }},\r\n {L\"FindFirstFileEx (Large Fetch)\", [&](std::vector<FileInfo>& files) {\tBenchmarkFindFirstFileEx(directory, files, FindExInfoStandard, FIND_FIRST_EX_LARGE_FETCH);\r\n }},\r\n {L\"GetFileAttributesEx\", [&](std::vector<FileInfo>& files) {\r\n BenchmarkGetFileAttributesEx(directory, files);\r\n }},\r\n {L\"std::filesystem\", [&](std::vector<FileInfo>& files) {\r\n BenchmarkStdFilesystem(directory, files);\r\n }},\r\n {L\"GetFileInformationByHandleEx\", [&](std::vector<FileInfo>& files) {\r\n BenchmarkGetFileInformationByHandleEx(directory, files);\r\n }}\r\n};\r\n\r\nstd::vector<std::pair<std::wstring, double>> results;\r\n\r\nfor (const auto& benchmark : benchmarks) {\r\n std::vector<FileInfo> files;\r\n files.reserve(2000); \/\/ Reserve space outside the timing measurement\r\n\r\n auto start = std::chrono::high_resolution_clock::now();\r\n benchmark.second(files);\r\n auto end = std::chrono::high_resolution_clock::now();\r\n\r\n std::chrono::duration<double> elapsed = end - start;\r\n results.emplace_back(benchmark.first, elapsed.count());\r\n}\r\n\r\nPrintResultsTable(results);\r\n<\/code><\/pre>\nPerformance Results<\/h2>\n
Method Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0014831000 162.868\r\nFindFirstFileEx (Standard) 0.0014817000 163.022\r\nFindFirstFileEx (Large Fetch) 0.0011792000 204.842\r\nGetFileAttributesEx 0.2415497000 1.000\r\nstd::filesystem 0.0609313000 3.964\r\nGetFileInformationByHandleEx 0.0044168000 54.689\r\n\r\n\/\/ second run:\r\nMethod Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0013805000 44.947\r\nFindFirstFileEx (Standard) 0.0011310000 54.863\r\nFindFirstFileEx (Large Fetch) 0.0009071000 68.404\r\nGetFileAttributesEx 0.0616772000 1.006\r\nstd::filesystem 0.0620496000 1.000\r\nGetFileInformationByHandleEx 0.0025246000 24.578\r\n<\/code><\/pre>\nMethod Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0014455000 150.287\r\nFindFirstFileEx (Standard) 0.0015029000 144.547\r\nFindFirstFileEx (Large Fetch) 0.0012086000 179.745\r\nGetFileAttributesEx 0.2172402000 1.000\r\nstd::filesystem 0.0609186000 3.566\r\nGetFileInformationByHandleEx 0.0025069000 86.657\r\n\r\nMethod Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0012020000 50.908\r\nFindFirstFileEx (Standard) 0.0011614000 52.688\r\nFindFirstFileEx (Large Fetch) 0.0008887000 68.856\r\nGetFileAttributesEx 0.0611920000 1.000\r\nstd::filesystem 0.0611760000 1.000\r\nGetFileInformationByHandleEx 0.0025835000 23.686\r\n<\/code><\/pre>\nMethod Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0077623000 84.975\r\nFindFirstFileEx (Standard) 0.0828258000 7.964\r\nFindFirstFileEx (Large Fetch) 0.0144611000 45.612\r\nGetFileAttributesEx 0.6595977000 1.000\r\nstd::filesystem 0.3022779000 2.182\r\nGetFileInformationByHandleEx 0.0051569000 127.906\r\n\r\nMethod Time (seconds) Speedup Factor\r\nFindFirstFileEx (Basic) 0.0069814000 43.844\r\nFindFirstFileEx (Standard) 0.0148472000 20.616\r\nFindFirstFileEx (Large Fetch) 0.0140663000 21.761\r\nGetFileAttributesEx 0.3060932000 1.000\r\nstd::filesystem 0.3011346000 1.016\r\nGetFileInformationByHandleEx 0.0051614000 59.304\r\n<\/code><\/pre>\nFindFirstFileEx<\/code> with the Standard<\/code> flag was the fastest method in uncached scenarios, offering speedups up to 129x compared to GetFileAttributesEx<\/code>. However, in cached scenarios, FindFirstFileEx<\/code> (Basic and Standard) achieved over 50x speedup improvements. The parameters for “Large Fetch” seems to increase the performance.<\/p>\nFindFirstFileEx<\/code> (Basic) demonstrated a speedup factor of over 179x in the first run and went down to 68 in the second run. In the directory with 5000 files, we can see that GetFileInformationByHandleEx<\/code> takes crown and acheives 59x speedup, while other techniques reaches 43x max. Notably, std::filesystem<\/code> performed on par with GetFileAttributesEx<\/code> .<\/p>\nFurther Techniques<\/h2>\n
GetFileAttributes[Ex]<\/code> using the same techniques as this article. But Visual Assist also improved performance through further techniques. My simple benchmark showed 50x speedups, but we cannot directly compare it with the final Visual Assist, as the tool does many more things with files.<\/p>\nSummary<\/h2>\n
FindFirstFileEx<\/code> or via GetFileInformationByHandleEx<\/code>. So if you want to do this operation hundreds of times, it’s best to measure time and choose the best technique. What’s more, if you expect to have lots of files in a directory it’s good to check techniques offering larger buffers.<\/p>\nfilesystem<\/code> library offer a robust and standardized way to work with files and directories, it can be limited in terms of performance. In many cases, if you need super optimal performance, you need to open the hood and work with the specific operating system API.<\/p>\nBack to you<\/h4>\n
\n