欢迎来到 嗅灵易学

零基础也能上手的脚本技术课,一对一答疑带你入门

[翻译]《深入解析windows操作系统第6版下册》第10章:内存管理(第二部分)

[翻译]《深入解析windows操作系统第6版下册》第10章:内存管理(第二部分)

第二部分预计翻译的内容如下:


由于第二及其后续部分的多数实验涉及内核调试,因此掌握WinDbg与KD.exe等调试器的使用对上机实践会很有帮助。
下面是原文+译文

Kernel-Mode Heaps (System Memory Pools)
At system initialization, the memory manager creates two dynamically sized memory pools, or heaps, that most kernel-mode components use to allocate system memory:
Nonpaged pool Consists of ranges of system virtual addresses that are guaranteed to reside in physical memory at all times and thus can be accessed at any time without incurring a page fault; therefore, they can be accessed from any IRQL. One of the reasons nonpaged pool is required is because of the rule described in Chapter 2 in Part 1: page faults can’t be satisfied at DPC/dispatch level or above. Therefore, any code and data that might execute or be accessed at or above DPC/dispatch level must be in nonpageable memory.
Paged pool A region of virtual memory in system space that can be paged into and out of the system. Device drivers that don’t need to access the memory from DPC/dispatch level or above can use paged pool. It is accessible from any process context.

内核模式堆(系统内存池)
系统初始化时,内存管理器创建2种可动态调整大小的内存池或堆,多数内核模式组件使用它们来分配系统内存:
■   不可分页池   由一系列在任何时候都能保证驻留物理内存中的系统虚拟地址组成,从而可被随时访问,不会引发一个缺页异常;因此它们可经由任意IRQL(中断请求级别)访问。需要非分页池的其中一个原因在于本书上册第2章论述的法则:DPC/dispatch(延迟过程调用/分发)或之上级别的中断不能满足缺页异常。因此,任何DPC/dispatch或其上级别中断可能执行的代码或被访问的数据必须位于不可分页池中。
■   可分页池   系统空间中的一个虚拟内存区域,可被换入和换出系统。不需要从DPC/dispatch或其上级别访问内存的设备驱动程序可以使用可分页池。可分页池能够从任何进程上下文进行访问。

Both memory pools are located in the system part of the address space and are mapped in the virtual address space of every process. The executive provides routines to allocate and deallocate from these pools; for information on these routines, see the functions that start with ExAllocatePool and ExFreePool in the WDK documentation.
这2种内存池都位于地址空间的“系统部分”,并且被映射到每个进程的虚拟地址空间。执行体提供了从这些池中分配和释放内存的例程;在WDK(Windows驱动程序开发工具)的文档中查看以 ExAllocatePool 和 ExFreePool 名称开头的函数,可以获取这些例程的更多信息。
Systems start with four paged pools (combined to make the overall system paged pool) and one nonpaged pool; more are created, up to a maximum of 64, depending on the number of NUMA nodes on the system. Having more than one paged pool reduces the frequency of system code blocking on simultaneous calls to pool routines. Additionally, the different pools created are mapped
across different virtual address ranges that correspond to different NUMA nodes on the system. (The different data structures, such as the large page look-aside lists, to describe pool allocations are also mapped across different NUMA nodes. More information on NUMA optimizations will follow later.)

初始状态下,系统有4个可分页池(它们结合在一起构建出整体的系统可分页池)和一个不可分页池;并且可以创建更多的池,最多64个,这取决于系统上的NUMA节点数。拥有一个以上的可分页池能够减少系统代码阻塞对池中例程的同时调用的频率。此外,创建的不同类型池被映射跨越不同的虚拟地址范围,这些池对应于系统上不同的NUMA节点。(不同的数据结构,例如用于描述池分配额的大页面后备列表,也被映射跨越不同的NUMA节点。稍后将讨论与NUMA优化相关的更多内容。)
In addition to the paged and nonpaged pools, there are a few other pools with special attributes or uses. For example, there is a pool region in session space, which is used for data that is common to all processes in the session. (Sessions are described in Chapter 1 in Part 1.) There is a pool called, quite literally, special pool. Allocations from special pool are surrounded by pages marked as no-access to help isolate problems in code that accesses memory before or after the region of pool it allocated. Special pool is described in Chapter 14.
除了分页与非分页池外,还有一些其它带有特殊属性或用途的池。例如,在会话空间中有一个池区,作为(存储)该会话中所有进程共用的数据。(本书上册第1章介绍了与会话相关的概念。)
有一个被称为“特殊池”的池(正如其字面名称special pool所示)。从special pool中的内存分配围绕着被标记为不可访问(no-access)的页面进行,(译注:请参考第一部分译文的表10-2)这有
助于找出代码中的问题:访问了被分配池区之前或之后的内存。(译注:推测是由于这些尚未被作为池区分配的内存页还处于不可访问的保护状态)第14章将讨论特殊池。(译注:因为这个系列仅翻译第10章,各位有兴趣可以自行翻译第14章与特殊池相关的内容来了解)

Pool Sizes
Nonpaged pool starts at an initial size based on the amount of physical memory on the system and then grows as needed. For nonpaged pool, the initial size is 3 percent of system RAM. If this is less than 40 MB, the system will instead use 40 MB as long as 10 percent of RAM results in more than 40 MB; otherwise 10 percent of RAM is chosen as a minimum.
Windows dynamically chooses the maximum size of the pools and allows a given pool to grow from its initial size to the maximums shown in Table 10-4.

池大小
不可分页池以一个基于系统物理内存总量的初始大小为起点,并且能够按需增大。对于不可分页池,初始大小为系统RAM的 3%。如果据此计算出来的值小于40MB,只要RAM的 10%超过40MB,那么系统就会使用40MB来取代该值;否则将选取RAM的 10%作为最小值(下限)。Windows会动态选择池的最大值(上限),并且允许一个给定的池从自身的初始大小开始增大,乃至上限,如下表10-4所示:


Four of these computed sizes are stored in kernel variables, three of which are exposed as performance counters, and one is computed only as a performance counter value. These variables and counters are listed in Table 10-5.
其中4个计算出的大小存储在相应的内核变量中,其中的3个通过性能计数器对外暴露,剩余的仅作为一个性能计数器值来计算。下表10-5列出了这些变量与计数器:


EXPERIMENT: Determining the Maximum Pool Sizes
You can obtain the pool maximums by using either Process Explorer or live kernel debugging (explained in Chapter 1 in Part 1). To view pool maximums with Process Explorer, click on View, System Information, and then click the Memory tab. The pool limits are displayed in the Kernel Memory middle section, as shown here:

实验: 确定池大小上限
您可以通过使用进程浏览器或者实时内核调试(在本书上册第1章中介绍),获取池上限。(译注:实时内核调试是指,不需要将系统配置成以调试模式引导的内核运行时调试,这需要sysinternals套件中的LiveKd工具结合WinDbg.exe或Kd.exe;后者是命令行下的内核调试器,限于篇幅,具体使用方法请参考上册第1章。)要使用进程浏览器查看池上限,在其主界面菜单栏上点击“View”->“System Information”,在弹出的窗口中切换到“Memory”选项卡。在Kernel Memory 栏位的中间部分显示出可分页与不可分页池上限的信息,如下图所示:


Note that for Process Explorer to retrieve this information, it must have access to the symbols for the kernel running on your system. (For a description of how to configure Process Explorer to use symbols, see the experiment “Viewing Process Details with Process Explorer” in Chapter 1 in Part 1.)
需要注意:在您的系统上,进程浏览器为了检索这个信息(上限 ),它必须能够访问用于注解内核运行时行为的符号文件。(参见本书上册第1章中的实验“通过进程浏览器查看进程的详细信息”来了解如何配置进程浏览器使用符号)(译注:如果没有配置符号文件,那么将无法检索到池上限,如下图所示,是在我的真实机器上测试的结果:)

To view the same information by using the kernel debugger, you can use the !vm command as shown here:
要通过使用内核调试器查看相同的信息,您可以使用 !vm 命令,如下所示:

[SIZE="4"][FONT="微软雅黑"]kd> !vm

1: kd> !vm

*** Virtual Memory Usage ***

Physical Memory: 851757 ( 3407028 Kb)

Page File: \??\C:\pagefile.sys

Current: 3407028 Kb Free Space: 3407024 Kb

Minimum: 3407028 Kb Maximum: 4193280 Kb

Available Pages: 699186 ( 2796744 Kb)

ResAvail Pages: 757454 ( 3029816 Kb)

Locked IO Pages: 0 ( 0 Kb)

Free System PTEs: 370673 ( 1482692 Kb)

Modified Pages: 9799 ( 39196 Kb)

Modified PF Pages: 9798 ( 39192 Kb)

NonPagedPool Usage: 0 ( 0 Kb)

NonPagedPoolNx Usage: 8735 ( 34940 Kb)

NonPagedPool Max: 522368 ( 2089472 Kb)

PagedPool 0 Usage: 17573 ( 70292 Kb)

PagedPool 1 Usage: 2417 ( 9668 Kb)

PagedPool 2 Usage: 0 ( 0 Kb)

PagedPool 3 Usage: 0 ( 0 Kb)

PagedPool 4 Usage: 28 ( 112 Kb)

PagedPool Usage: 20018 ( 80072 Kb)

PagedPool Maximum: 523264 ( 2093056 Kb)[/FONT][/SIZE]

On this 4-GB, 32-bit system, nonpaged and paged pool were far from their maximums.
上面来自于一个4GB物理内存的32位系统的输出表明,不可分页池与可分页池的当前值还远未到达上限。(译注:参见表10-4,32位系统上的可分页池上限为2GB,与内核调试器的输出,即圆括号中的 2093056Kb ,大致相符,而32位系统的不可分页池上限,取“物理内存的75%与2GB”两者中的较小值,例子中是4GB物理内存,因此75%为3GB,因此上限取2GB,也与输出相符。下图则是我在32位Windows 7旗舰版真实机器上的测试结果:)


You can also examine the values of the kernel variables listed in Table 10-5. The following were taken from a 32-bit system:
您也可以通过内核调试器审视在表10-5中列出的内核变量值。下面的输出同样取自一个32位系统:
[FONT="微软雅黑"][SIZE="4"]lkd> ? poi(MmMaximumNonPagedPoolInBytes)

Evaluate expression: 2139619328 = 7f880000

lkd> ? poi(MmSizeOfPagedPoolInBytes)

Evaluate expression: 2143289344 = 7fc00000[/SIZE][/FONT]

From this example, you can see that the maximum size of both nonpaged and paged pool is approximately 2 GB, typical values on 32-bit systems with large amounts of RAM. On the system used for this example, current nonpaged pool usage was 35 MB and paged pool usage was 80 MB, so both pools were far from full.
从这个例子中您可以看到,不可分页池与可分页池的最大上限都大约为2GB,这是拥有大量RAM的32位系统上的典型值。在例子中的这个系统上,当前的非分页池使用率为35MB,可分页池使用率为80MB,因此两者都远未到达上限。(译注:我的测试结果如下:)


Monitoring Pool Usage
The Memory performance counter object has separate counters for the size of nonpaged pool and paged pool (both virtual and physical). In addition, the Poolmon utility (in the WDK) allows you to monitor the detailed usage of nonpaged and paged pool. When you run Poolmon, you should see a display like the one shown in Figure 10-6.

监视池的使用情况
内存性能计数器对象有用于计算不可分页池与可分页池大小的独立计数器(包括虚拟的和物理的)。此外,Poolmon实用程序(包含在WDK中)允许您监测不可分页池与可分页池的详细使用情况。运行Poolmon时,您应该会看到如下图10-6所示的界面:(译注:WDK可以在MSDN网站上搜索下载,当前最新版本为WDK10。原文提供的图片没有体现出高亮功能,因此我用自己机器上的截图代替)


The highlighted lines you might see represent changes to the display. (You can disable the highlighting feature by typing a slash (/) while running Poolmon. Type / again to reenable highlighting.)
Type ? while Poolmon is running to bring up its help screen. You can configure which pools you want to monitor (paged, nonpaged, or both) and the sort order. For example, by pressing the P key until only nonpaged allocations are shown, and then the D key to sort by the Diff (differences) column, you can find out what kind of structures are most numerous in nonpaged pool. Also, the command-line options are shown, which allow you to monitor specific tags (or every tag but one tag). For example, the command poolmon –iCM will monitor only CM tags (allocations from the configuration manager,
which manages the registry). The columns have the meanings shown in Table 10-6.

您可能会看到高亮突出的行,它代表显示中的变更。(您可以通过在运行Poolmon时,输入一个正斜杠[/],来禁用高亮功能。再次输入 / 可以重新启用高亮。)输入 ? 则带出Poolmon的帮助信息。
您可以配置要监视的池(可分页/不可分页,或监视两者)并进行排序。举例来说,通过持续按住P键直到仅显示不可分页池的分配情况,然后再按下D键通过Diff(差异)列排序,您可以找出不可分页池中数量最多的结构类型。再者,显示的命令行选项允许您监控特定的标签(或排除特定标签)。例如,命令 poolmon –iCM 将仅监视CM标签(分配给配置管理器的系统内存池,其管理注册表)。Poolmon中每个列的含义如下表10-6所示:



为了帮助大家进一步理解这些列的含义,请参见下图的实践分析:

For a description of the meaning of the pool tags used by Windows, see the file \Program Files\Debugging Tools for Windows\Triage\Pooltag.txt. (This file is installed as part of the Debugging Tools for Windows, described in Chapter 1 in Part 1.) Because third-party device driver pool tags are not listed in this file, you can use the –c switch on the 32-bit version of Poolmon that comes with the WDK
to generate a local pool tag file (Localtag.txt). This file will contain pool tags used by drivers found on your system, including third-party drivers. (Note that if a device driver binary has been deleted after it was loaded, its pool tags will not be recognized.)

对于Windows使用的池标签含义,C:\Program Files\Debugging Tools for Windows\Triage\Pooltag.txt  这个文件给出了详细的解释。(此文件作为Windows调试工具,即SDK的一部分安装,本书上册第1章讨论了SDK与相关的内容)(译注:该文件也会作为WDK的一部分安装,
路经为X:\Windows Kits\10\Debuggers\x86\triage\Pooltag.txt,X为您安装WDK时指定的驱动器盘符,并且WDK集成了SDK,例如,Kd.exe与WinDbg.exe就位于前述路径的x86子目录中,换言之,安装WDK为进行内核调试相关的一站式解决方案)
由于Pooltag.txt中没有列出第三方设备驱动程序的池标签,对于WDK中的32位版 Poolmon.exe,你可以使用 -c 开关,来生成一个本地池标签文件(Localtag.txt)。该文件将包含被你的系统上所有驱动程序使用的池标签,包括由第三方驱动使用的。(注意,如果一个设备驱动程序的二进制文件在被加载进内核后删除,Poolmon将无法识别其池标签。)
(译注:下面节录了Pooltag.txt的前面几行,我们可以在Poolmon的输出中复制要查找的池标签名称,然后粘贴在该文件的搜索对话框[ctrl + F]中进行查询:)

[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]//

//    Copyright (C) Microsoft.  All rights reserved.

//

rem

rem    Pooltag.txt

rem

rem    This file lists the tags used for pool allocations by kernel mode components

rem    and drivers.

rem

rem    The file has the following format:

rem       <PoolTag> - <binary-name> - <Description>

rem

rem    Pooltag.txt is installed with Debugging Tools for Windows (in %windbg%\triage)

rem    and with the Windows DDK (in %winddk%\tools\other\platform\poolmon, where

rem    platform is amd64, i386, or arm).

rem

@GMM - <unknown>    - (Intel video driver) Memory manager

@KCH - <unknown>    - (Intel video driver) Chipset specific service

@MP  - <unknown>    - (Intel video driver) Miniport related memory

@SB  - <unknown>    - (Intel video driver) Soft BIOS

_ATI - <unknown>    - ATI video driver

_LCD - monitor.sys  - Monitor PDO name buffer

8042 - i8042prt.sys - PS/2 keyboard and mouse

AdSv - vmsrvc.sys   - Virtual Machines Additions Service

ARPC - atmarpc.sys  - ATM ARP Client[/COLOR][/SIZE][/FONT]

(该文件描述了池标签(对应的内存池)是分配给那种内核模式组件或设备驱动程序使用的,例如 @GMM 表示Intel 视频驱动的内存管理器组件使用的池; 8042 表示PS/2 接口键盘和鼠标驱动程序 i8042prt.sys使用的池。。。以此类推。)
Alternatively, you can search the device drivers on your system for a pool tag by using the Strings.exe tool from Sysinternals. For example, the command
strings %SYSTEMROOT%\system32\drivers\*.sys | findstr /i "abcd"
will display drivers that contain the string “abcd”. Note that device drivers do not necessarily have to be located in %SystemRoot%\System32\Drivers—they can be in any folder. To list the full path of all loaded drivers, open the Run dialog box from the Start menu, and then type Msinfo32. Click Software Environment, and then click System Drivers. As already noted, if a device driver has been loaded and then deleted from the system, it will not be listed here.

或者,你可以通过Sysinternals的String.exe工具,搜索系统上的设备驱动程序,来获取一个池标签。例如,下面命令:
[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]strings %SYSTEMROOT%\system32\drivers\*.sys | findstr /i "abcd"[/COLOR][/SIZE][/FONT]

将显示包含字符串“abcd”的驱动程序。注意,设备驱动程序不一定非得位于 %SystemRoot%\System32\Drivers  目录——它们可以位于任何文件夹。要列出所有已加载驱动的完整路径,首先用 win + R 组合键从开始菜单中打开运行对话框,输入 Msinfo32。点击并展开主界面左侧的“软件环境”节点,然后单击“系统驱动程序”。如前文所述,如果一个已加载驱动的二进制文件被从磁盘上删除,将不会显示在列表中。
An alternative to view pool usage by device driver is to enable the pool tracking feature of Driver Verifier, explained later in this chapter. While this makes the mapping from pool tag to device driver unnecessary, it does require a reboot (to enable Driver Verifier on the desired drivers). After rebooting with pool tracking enabled, you can either run the graphical Driver Verifier Manager (%SystemRoot%\System32\Verifier.exe) or use the Verifier /Log command to send the pool usage information to a file.
另一种查看设备驱动程序的池使用情况的方法是,启用驱动程序验证器的池跟踪功能,本章稍后解释。尽管这没有必要进行从池标签到设备驱动程序的映射,它确实需要重启(以对目标驱动启用驱动程序验证器)。启用池跟踪功能并重启后,你可以运行图形界面的驱动程序验证器管理器(%SystemRoot%\System32\Verifier.exe)或者使用该验证器的 /Log  命令将池使用率信息发送到一个文件。
Finally, you can view pool usage with the kernel debugger !poolused command. The command !poolused 2 shows nonpaged pool usage sorted by pool tag using the most amount of pool. The command !poolused 4 lists paged pool usage, again sorted by pool tag using the most amount of pool. The following example shows the partial output from these two commands:
最后,你可以使用内核调试器的  !poolused  命令查看池使用率。!poolused 2 命令按照池的使用量来排序属于不可分页池的池标签; !poolused 4  命令按照池的使用量来排序属于可分页池的池标签。下面的例子展示这2个命令的部分输出:(译注:从输出中我们可以发现,某些内核模式组件或设备驱动程序对应的池标签既在不可分页池中分配了内存,也在可分页池中分配了内存,如 Etw 缓冲区。尽管原文没有对输出结果深入讨论,我们可以自行测试)



要进一步使用Poolmon监控内核内存,下面2个MSDN链接提供了丰富的信息:
https://msdn.microsoft.com/zh-cn/library/windows/hardware/ff550442%28v=vs.85%29.aspx
https://msdn.microsoft.com/zh-cn/library/windows/hardware/ff560135%28v=vs.85%29.aspx

EXPERIMENT: Troubleshooting a Pool Leak
In this experiment, you will fix a real paged pool leak on your system so that you can put to use the techniques described in the previous section to track down the leak. The leak will be generated by the Notmyfault tool from Sysinternals. When you run Notmyfault.exe, it loads the device driver Myfault.sys and presents the following dialog box:
1. Click the Leak tab, ensure that Leak/Second is set to 1000 KB, and click the Leak Paged button. This causes Notmyfault to begin sending requests to the Myfault device driver to allocate paged pool. Notmyfault will continue sending requests until you click the Stop Paged button. Note that paged pool is not normally released even when you close a program that has caused it to occur (by interacting with a buggy device driver); the pool is permanently leaked until you reboot the system. However, to make testing easier, the Myfault device driver detects that the process was closed and frees its allocations.
2. While the pool is leaking, first open Task Manager and click on the Performance tab. You should notice Kernel Memory (MB): Paged climbing. You can also check this with Process Explorer’s System Information display. (Click View, System Information, and then the Memory tab.)
3. To determine the pool tag that is leaking, run Poolmon and press the B key to sort by the number of bytes. Press P twice so that Poolmon is showing only paged pool.
You should notice the pool tag “Leak” climbing to the top of the list. (Poolmon shows
changes to pool allocations by highlighting the lines that change.)
4. Now press the Stop Paged button so that you don’t exhaust paged pool on your system.
5. Using the technique described in the previous section, run Strings (from Sysinternals) to look for driver binaries that contain the pool tag “Leak”:
Strings %SystemRoot%\system32\drivers\*.sys | findstr Leak
This should display a match on the file Myfault.sys, thus confirming it as the driver using the “Leak” pool tag.

实验:排查内存池泄露
在本实验中,你将修复系统上一个真实的可分页池泄露问题,以便你能够运用本章前面部分讨论的技能来追查出该泄露。通过Sysinternals的Notmyfault工具产生泄露。运行Notmyfault.exe时,它加载Myfault.sys设备驱动程序并且显示下面的对话框:
(译注:在https://technet.microsoft.com/en-us/sysinternals/bb963901  有提供 Notmyfault工具下载的链接)

1.   单击Leak标签,确保Leak/Second设置成1000KB,然后单击Leak Paged按钮。这会导致Notmyfault开始向Myfault.sys设备驱动程序发送请求来分配可分页池。Notmyfault将持续发送请求,直到你点击Stop Paged按钮。注意,甚至在你关闭导致发生泄漏的Notmyfault.exe程序(通过与Myfault.sys这个古怪的设备驱动交互)后,可分页池中已分配的内存通常也不会得到释放;它会长久性地泄漏,直到你重启系统。然而,为了方便用户测试,Myfault.sys若检测到Notmyfault进程退出,它会释放分配的空间。
2.   在泄漏过程中,首先打开任务管理器并且单击性能标签。你应该关注“核心内存(MB)”栏位中“分页数”的值攀升。你也可以通过进程浏览器的“系统信息”界面(热键为Ctrl + I)进行相同的检查。(View -> System Informationt,然后切换到 Memory 标签)
3.   要确定正在泄露的池标签,运行Poolmon(在CMD命令行提示符下)然后按下 B 键通过字节数排序。按2次 P 键仅显示可分页池。现在你应该关注列表顶端的“Leak”池标签这一行的相应数值攀升。(Poolmon通过高亮标记发生变化的行来反映与池分配相关的变更)
4.   现在按下Stop Paged按钮,避免耗尽你系统上的内核可分页池空间。(译注:默认每秒泄漏/消耗1MB,还是很快的)
5.   使用本章前面部分讨论的技巧,运行Sysinternals的Strings工具,查找包含池标签“Leak”的驱动程序二进制文件:
[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]Strings %SystemRoot%\system32\drivers\*.sys | findstr Leak[/COLOR][/SIZE][/FONT]

这应该会显示Myfault.sys文件中的匹配项,从而证实它是使用““Leak”池标签的驱动。

Look-Aside Lists
Windows also provides a fast memory allocation mechanism called look-aside lists. The basic difference between pools and look-aside lists is that while general pool allocations can vary in size, a look-aside list contains only fixed-sized blocks. Although the general pools are more flexible in terms of what they can supply, look-aside lists are faster because they don’t use any spinlocks.

后备列/链表
Windows也提供了一种快速的内存分配机制,叫做后备列/链表。池与后备列/链表的根本区别在于,一般的池分配大小可变,而一个后备列/链表仅包含固定大小的块。虽然通用池就它们能够提供的内存空间而言明显更灵活,但是后备列/链表更快速,因为它们没有使用任何自旋锁。

Executive components and device drivers can create look-aside lists that match the size of frequently allocated data structures by using the ExInitializeNPagedLookasideList and ExInitializePagedLookasideList functions (documented in the WDK). To minimize the overhead of multiprocessor synchronization, several executive subsystems (such as the I/O manager, cache manager, and object manager) create separate look-aside lists for each processor for their frequently accessed data structures. The executive also creates a general per-processor paged and nonpaged look-aside list for small allocations (256 bytes or less).
Windows执行体组件以及设备驱动程序可以通过使用ExInitializeNPagedLookasideList 和 ExInitializePagedLookasideList函数(在WDK中文档化),创建后备列/链表,其大小匹配常见分配的数据结构。为了最小化多处理器同步方面的开销,一些执行体子系统(诸如I/O管理器,缓存管理器,以及对象管理器)会为每个处理器频繁访问的数据结构创建单独的后备列/链表。执行体也为每个处理器分别创建一个普通的可分页和不可分页后备列/链表,用来分配256字节或更小的内存空间。
If a look-aside list is empty (as it is when it's first created), the system must allocate from paged or nonpaged pool. But if it contains a freed block, the allocation can be satisfied very quickly. (The list grows as blocks are returned to it.) The pool allocation routines automatically tune the number of freed buffers that look-aside lists store according to how often a device driver or executive subsystem allocates from the list—the more frequent the allocations, the more blocks are stored on a list.
Look-aside lists are automatically reduced in size if they aren’t being allocated from. (This check happens once per second when the balance set manager system thread wakes up and calls the function ExAdjustLookasideDepth.)

如果一个后备列/链表为空(当首次创建时就是如此),系统必须从可分页或不可分页池中分配。而如果它包含一个已释放的块,则可以非常迅速地满足分配需求。(返回的列表成块增长)
池分配例程自动调整后备列/链表中存储的已释放缓冲的数量,这根据一个设备驱动程序或执行体子系统申请从列/链表中分配内存的频率而定——分配得越频繁,列/链表中存储的已释放块就越多。
如果它们没有被从中分配,后备列/链表会自动缩小尺寸。(当平衡集管理器系统线程被唤醒,并调用函数ExAdjustLookasideDepth时,这种检查每秒钟发生一次)(译著:关于平衡集管理器系统线程,请参考第一部分译文“内存管理器的组件”小节)

EXPERIMENT: Viewing the System Look-Aside Lists
You can display the contents and sizes of the various system look-aside lists with the kernel debugger !lookaside command. The following excerpt is from the output of this command:

实验:查看系统的后备列/链表
你可以通过内核调试器指令 !lookaside ,显示各种系统后备列/链表的内容和大小。下面摘录了该命令的部分输出:
(译注:原文没有对其输出进行解释,因此下面仅给出真实机器上的调试结果供参考,可以通过WDK文档,web搜索引擎挖掘输出中各种参数,字段的含义,就留给各位当作家庭作业好了)


Heap Manager
Most applications allocate smaller blocks than the 64-KB minimum allocation granularity possible using page granularity functions such as VirtualAlloc and VirtualAllocExNuma. Allocating such a large area for relatively small allocations is not optimal from a memory usage and performance standpoint.
To address this need, Windows provides a component called the heap manager, which manages allocations inside larger memory areas reserved using the page granularity memory allocation functions. The allocation granularity in the heap manager is relatively small: 8 bytes on 32-bit systems, and 16 bytes on 64-bit systems. The heap manager has been designed to optimize memory usage and performance in the case of these smaller allocations.

堆管理器
多数应用程序分配比64KB最小分配粒度更小的块,可能使用页面粒度函数如VirtualAlloc 和 VirtualAllocExNuma。相对于小范围分配而言,分配这样大片的一个区域从内存利用和性能角度来看,并不是最佳方案。(译注:即便使用了页面粒度函数,从应用程序级别的一般分配粒度下限64KB到单页粒度4KB,都算分配的大内存区域,请参考第一部分译文“分配粒度”小节)
为满足这一需求,Windows提供了一个叫做堆管理器的组件,它管理较大内存区域内的(小范围)分配,并且保留对页面粒度内存分配函数的使用。堆管理器支持的分配粒度相对较小:在32位系统上为8字节(一个“四字”);在64位系统上为16字节。堆管理器被设计成用来优化这些小范围分配场景下的内存利用和性能。

The heap manager exists in two places: Ntdll.dll and Ntoskrnl.exe. The subsystem APIs (such as the Windows heap APIs) call the functions in Ntdll, and various executive components and device drivers call the functions in Ntoskrnl. Its native interfaces (prefixed with Rtl) are available only for use in internal Windows components or kernel-mode device drivers. The documented Windows API interfaces to the heap (prefixed with Heap) are forwarders to the native functions in Ntdll.dll. In addition, legacy APIs (prefixed with either Local or Global) are provided to support older Windows applications, which also internally call the heap manager, using some of its specialized interfaces to support legacy behavior. The C runtime (CRT) also uses the heap manager when using functions such as malloc, free, and the C++ new operator. The most common Windows heap functions are:
■ HeapCreate or HeapDestroy Creates or deletes, respectively, a heap. The initial reserved and committed size can be specified at creation.
■ HeapAlloc Allocates a heap block.
■ HeapFree Frees a block previously allocated with HeapAlloc.
■ HeapReAlloc Changes the size of an existing allocation (grows or shrinks an existing block).
■ HeapLock or HeapUnlock Controls mutual exclusion to the heap operations.
■ HeapWalk Enumerates the entries and regions in a heap.
堆管理器存在于两处:Ntdll.dll 与Ntoskrnl.exe中。子系统(DLL)中的API函数 (例如Windows堆API函数)调用Ntdll中的函数,而各种执行体组件和设备驱动程序则调用Ntoskrnl中的函数。
堆管理器的原生接口(带有 Rtl 前缀)仅适用于内部使用的Windows组件或者内核模式设备驱动程序。已文档化的Windows堆API函数(带有 Heap 前缀)会转而调用Ntdll.dll中的原生函数。此外,传统(旧版)API函数(带有Local 或者 Global前缀)用来提供对较早Windows应用程序的支持,这些函数在其内部同样调用堆管理器,使用后者的一些专用接口以支持传统行为。C运行时库(CRT)中的函数,例如malloc,free,以及C++的 new 操作符(运算符)同样使用堆管理器提供的服务(接口)。最普遍使用的Windows堆函数有:
■ HeapCreate 或 HeapDestroy        分别创建和删除一个堆。在创建时可以指定初始的保留和提交大小(译注:关于保留和提交的内存,请参考第一部分译文相关小节)
■ HeapAlloc   分配一个“堆块”。
■ HeapFree    释放一个先前通过HeapAlloc分配的块。
■ HeapReAlloc   改变一个已分配块的大小(增大或减小一个现有的块)。
■ HeapLock 或 HeapUnlock   控制互斥的堆操作。
■ HeapWalk   枚举一个堆中的项和区域。

Types of Heaps
Each process has at least one heap: the default process heap. The default heap is created at process startup and is never deleted during the process’s lifetime. It defaults to 1 MB in size, but it can be made bigger by specifying a starting size in the image file by using the /HEAP linker flag. This size is just the initial reserve, however—it will expand automatically as needed. (You can also specify the initial committed size in the image file.)

堆类型
每个进程至少拥有一个堆:默认进程堆。它在进程启动时创建,并且在进程的生命周期内从不被删除。默认为1MB大小,可以通过使用 /HEAP 链接器标志在其映像文件中指定起始大小来增大。然而,这个指定的大小只是初始留出的大小——它将会按需自动扩大。(你也可以在映像文件中指定初始的提交大小。)

The default heap can be explicitly used by a program or implicitly used by some Windows internal functions. An application can query the default process heap by making a call to the Windows function GetProcessHeap. Processes can also create additional private heaps with the HeapCreate function.
When a process no longer needs a private heap, it can recover the virtual address space by calling HeapDestroy. An array with all heaps is maintained in each process, and a thread can query them with the Windows function GetProcessHeaps.

默认堆可以显式地被一个程序使用,或者隐式地被某些Windows内部函数使用。一个应用程序可以调用Windows函数GetProcessHeap,来查询其默认进程堆。进程也可以通过HeapCreate函数创建额外的私有堆。当一个进程不再需要一个私有堆,它可以通过调用HeapDestroy函数回收其虚拟地址空间。每个进程都维护了一个所有堆构成的数组,并且一个线程能够通过Windows函数GetProcessHeaps对其进行查询。
A heap can manage allocations either in large memory regions reserved from the memory manager via VirtualAlloc or from memory mapped file objects mapped in the process address space.
The latter approach is rarely used in practice, but it’s suitable for scenarios where the content of the blocks needs to be shared between two processes or between a kernel-mode and a user-mode component. The Win32 GUI subsystem driver (Win32k.sys) uses such a heap for sharing GDI and User objects with user mode. If a heap is built on top of a memory mapped file region, certain constraints apply with respect to the component that can call heap functions. First, the internal heap structures use pointers, and therefore do not allow remapping to different addresses in other processes. Second, the synchronization across multiple processes or between a kernel component and a user process is not supported by the heap functions. Also, in the case of a shared heap between user mode and kernel mode, the user-mode mapping should be read-only to prevent user-mode code from corrupting the heap’s internal structures, which would result in a system crash. The kernel-mode driver is also responsible for not putting any sensitive data in a shared heap to avoid leaking it to user mode.

堆能够管理的分配区域可以是由内存管理器保留(分配)的大片内存区域(通过VirtualAlloc函数),或者是映射到进程地址空间中的“内存映射文件对象”。(参见第一部分译文“共享内存与文件映射”小节)后一种方法在实践中很少使用,但它适用于下面场景:需要在两个进程间;或在一个内核模式组件与一个用户模式组件之间共享块的内容时。(换言之,堆能够将内存映射文件对象的大片区域进一步划分为以块为单位来管理共享)
Win32 GUI 子系统驱动程序(Win32k.sys)就使用了这样的一个堆,将GDI(图形设备接口)和User objects与用户模式进行共享。(译注:User objects暂译为用户对象;本书上册第2章提到,win32k.sys是windows子系统的内核模式组件[ csrss.exe是windows子系统的用户模式组件之一],win32k.sys可以看成是windows系统原生的“设备驱动程序”,其自身实现了两个重要的内核功能,即窗口管理器与图形设备接口,它们负责图形用户界面功能,比如对窗口的处理,用户界面控件,以及绘制等)
假设一个堆构建在一个内存映射文件区域的顶部,关于组件可以调用的堆函数方面将应用某些限制。首先,内部的堆结构使用指针,因此不允许重新映射到其它进程中的不同地址。其次,堆函数将不支持多进程间,或者一个内核组件与一个用户进程之间的同步。再者,就一个用户模式与内核模式间共享的堆来说,用户模式应该映射为只读,以防止用户模式代码损坏该堆的内部数据结构,从而导致系统崩溃。内核模式驱动程序也须负责不能够在一个共享堆中放置任何敏感数据,以避免将其泄露至用户模式。

Heap Manager Structure
As shown in Figure 10-7, the heap manager is structured in two layers: an optional front-end layer and the core heap. The core heap handles the basic functionality and is mostly common across the user-mode and kernel-mode heap implementations. The core functionality includes the management of blocks inside segments, the
management of the segments, policies for extending the heap, committing and decommitting memory, and management of the large blocks.

堆管理器结构
如下图10-7所示,堆管理器被结构化为2层:一个可选的前端层,以及核心堆。核心堆处理基本功能,并且是最为常见跨越用户与内核模式堆的实现。其核心功能包括段内块(blocks inside segments)的管理,段的管理,扩展堆的策略,提交和回收内存,以及大型块的管理。


For user-mode heaps only, an optional front-end heap layer can exist on top of the existing core functionality. The only front-end supported on Windows is the Low Fragmentation Heap (LFH). Only one front-end layer can be used for one heap at one time.
一个可选的前端堆层能够存在于现有核心功能的顶部(仅限于用户模式堆)。Windows支持的唯一前端层类型是低碎片堆(LFH)。一个堆一次只能使用一个前端层。
Heap Synchronization
The heap manager supports concurrent access from multiple threads by default. However, if a process is single threaded or uses an external mechanism for synchronization, it can tell the heap manager to avoid the overhead of synchronization by specifying HEAP_NO_SERIALIZE either at heap creation or on a per-allocation basis.

堆同步
堆管理器默认支持来自多个线程的并发访问。然而,如果一个进程是单线程的或者使用一个外部机制用于同步,就可以通过在堆创建时,或者在逐个分配堆块的基础上,指定HEAP_NO_SERIALIZE参数,来告知堆管理器避免进行同步导致的开销。

A process can also lock the entire heap and prevent other threads from performing heap operations for operations that would require consistent states across multiple heap calls. For instance, enumerating the heap blocks in a heap with the Windows function HeapWalk requires locking the heap if multiple threads can perform heap operations simultaneously.
一个进程也可以锁定整个堆从而防止其它线程对其执行堆操作,这些操作可能要求跨越多个堆调用间的一致状态。例如,假设多个线程能够同时执行堆操作,那么用Windows函数HeapWalk枚举堆中的堆块前需要锁定该堆。
If heap synchronization is enabled, there is one lock per heap that protects all internal heap structures. In heavily multithreaded applications (especially when running on multiprocessor systems), the heap lock might become a significant contention point. In that case, performance might be improved by enabling the front-end heap, described in an upcoming section.
如果启用了堆同步,每个堆有一把锁用来保护其内部所有的堆结构。在大量的多线程应用中(特别是当运行在一个多处理器系统上),堆锁可能会成为一个明显的竞争点。在这种情况下,通过启用前端堆层,性能或许能够得到改善,稍后将对其进行讨论。
The Low Fragmentation Heap
Many applications running in Windows have relatively small heap memory usage (usually less than 1 MB). For this class of applications, the heap manager’s best-fit policy helps keep a low memory footprint for each process. However, this strategy does not scale for large processes and multiprocessor machines. In these cases, memory available for heap usage might be reduced as a result of heap fragmentation. Performance can suffer in scenarios where only certain sizes are often used concurrently from different threads scheduled to run on different processors. This happens because several processors need to modify the same memory location (for example, the head of the look-aside list for that particular size) at the same time, thus causing significant contention for the corresponding cache line.

低碎片堆
运行在Windows上的许多应用程序有着相对较小的堆内存使用率(通常低于1MB)。对于此类应用,堆管理器的最适策略有助于为每个进程维持一个低内存占用。然而,该策略不能伸缩以适应大型进程和多处理器机器。在这种情况下,由于堆碎片的关系,可供堆使用的内存也许会减少。只有某些大小的堆块经常被“调度至不同处理器上运行的线程”同时使用,在这种情况下性能会变差。会发生这种情况是因为多个处理器需要在同一时间修改相同的内存位置(例如,该特定大小堆块的后备列/链表头部),从而导致对缓存内存内容的相应高速缓存行的明显竞争。(译注:推测这里应该是指对多核处理器共享的L3 cache 中的“高速缓存行”的竞争;因为每个处理器核有各自的L1  cache 与 L2 cache,所以不存在竞争,而L3 cache 存在竞争和缓存一致性等问题)

The LFH avoids fragmentation by managing allocated blocks in predetermined different block-size ranges called buckets. When a process allocates memory from the heap, the LFH chooses the bucket that maps to the smallest block large enough to hold the required size. (The smallest block is 8 bytes.)
The first bucket is used for allocations between 1 and 8 bytes, the second for allocations between 9 and 16 bytes, and so on, until the thirty-second bucket, which is used for allocations between 249 and 256 bytes, followed by the thirty-third bucket, which is used for allocations between 257 and 272 bytes, and so on. Finally, the one hundred twenty-eighth bucket, which is the last, is used for allocations between 15,873 and 16,384 bytes. (This is known as a binary buddy system.) Table 10-7 summarizes the different buckets, their granularity, and the range of sizes they map to.

低碎片堆(LFH)通过叫做buckets(桶)的单元来管理分配的堆块,系统为buckets预定了不同的块大小范围,使用buckets能够避免产生堆碎片。当一个进程请求从堆中分配内存时,LFH选取的bucket会映射到足以容纳所需(所请求)大小的最小的块。(最小的块为8字节。)第一个bucket用于(管理)所分配内存中的1~8字节,第二个bucket用于所分配内存中的9~16字节,以此类推,直到第32个bucket,其用于所分配内存中的249~256字节,紧随其后的第33个bucket用于所分配内存中的257~272字节,以此类推。
第128个bucket,也是最后一个,用于所分配内存中的15873~16384字节。(这种内存分配算法被称为“二进制伙伴系统”。)表10-7总结了不同的buckets,粒度(即单个bucket管理的字节数),及其映射的范围。


The LFH addresses these issues by using the core heap manager and look-aside lists. The Windows heap manager implements an automatic tuning algorithm that can enable the LFH by default under certain conditions, such as lock contention or the presence of popular size allocations that have shown better performance with the LFH enabled. For large heaps, a significant percentage of allocations is frequently grouped in a relatively small number of buckets of certain sizes. The allocation strategy used by LFH is to optimize the usage for these patterns by efficiently handling same-size blocks.
LFH通过使用核心堆管理器与后备列表来解决上述这些分配问题。Windows堆管理器实现了一个在某些条件下默认可以启用LFH的自动调整算法,例如锁竞争,或者在启用LFH的情况下,存在展现出更好性能的常用分配大小。对于大型堆,分配中占有一个显著比例的行为就是:在相对少数的某些大小buckets中频繁进行分组。LFH采用的分配策略是通过高效处理相同大小的块,来优化这些模式的使用。(译注:限于个人水平,本段翻译不当处还请提出)
To address scalability, the LFH expands the frequently accessed internal structures to a number of slots that is two times larger than the current number of processors on the machine. The assignment of threads to these slots is done by an LFH component called the affinity manager. Initially, the LFH
starts using the first slot for heap allocations; however, if a contention is detected when accessing some internal data, the LFH switches the current thread to use a different slot. Further contentions will spread threads on more slots. These slots are controlled for each size bucket to improve locality and minimize the overall memory consumption.

为解决可伸缩性,LFH将频繁访问的内部结构扩展为一系列的槽(slots),其数量为计算机当前处理器数目的2倍。一个叫做affinity manager(直译为“亲和管理器”)的LFH组件完成将线程分配给这些槽的工作。最初,LFH使用第一个槽进行堆分配;然而,如果在访问一些内部数据时检测到一个竞争,LFH会切换当前进程来使用一个不同的槽。进一步的竞争将传播线程到更多槽上。这些槽受控于每种大小的bucket,从而改善局部性,以及最小化总体的内存消耗。
Even if the LFH is enabled as a front-end heap, the less frequent allocation sizes may still continue to use the core heap functions to allocate memory, while the most popular allocation classes will be performed from the LFH. The LFH can also be disabled by using the HeapSetInformation API with the
HeapCompatibilityInformation class.

即便LFH作为前端堆层启用,不常见的分配大小仍可继续用于核心堆函数分配内存,而LFH将实施最常用的分配类型。也能够通过以HeapCompatibilityInformation类来使用HeapSetInformation函数(作为其第二个参数来调用,具体可以参考MSDN文档),从而禁用LFH。
(译注:本节的翻译和理解确实比较困难,各位可以进一步阅读 https://msdn.microsoft.com/zh-cn/library/windows/desktop/aa366750%28v=vs.85%29.aspx 的MSDN文档对低碎片堆的解释,在此引用一篇该文档的译文http://blog.csdn.net/wpc320/article/details/7949170)

Heap Security Features
As the heap manager has evolved, it has taken an increased role in early detection of heap usage errors and in mitigating effects of potential heap-based exploits. These measures exist to lessen the security effect of potential vulnerabilities in applications. The metadata used by the heap for internal management is packed with a high degree of randomization to make it difficult for an attempted exploit to patch the internal structures to prevent crashes or conceal the attack attempt. These blocks are also subject to an integrity check mechanism on the header to detect simple corruptions such as buffer overruns. Finally, the heap also uses a small degree of randomization of the base address (or handle). By using the HeapSetInformation API with the HeapEnableTerminationOnCorruption class, processes can opt in for an automatic termination in case of detected inconsistencies to avoid executing unknown code.

堆的安全功能
随着堆管理器的进化,它已在堆使用错误的早期检测,以及缓解潜在的基于堆的漏洞利用这两方面,发挥了日益重要的作用。这些措施的存在是为了减轻应用程序中潜在的漏洞带来的安全影响。
堆用于内部管理使用的元数据具有高度的随机性,如此一来,试图利用修补内部结构来防止崩溃或者隐藏攻击企图就变得很困难。这些堆块的头部也应用了完整性检查机制,从而能够检测到诸如缓冲区溢出等的简单损坏现象。最后,堆的基地址(或句柄)也使用了小程度的随机化。通过以HeapEnableTerminationOnCorruption类来使用HeapSetInformation函数,在检测到不一致的情况下,进程可以选择自动终止,以避免执行未知代码。

As an effect of block metadata randomization, using the debugger to simply dump a block header as an area of memory is not that useful. For example, the size of the block and whether it is busy or not are not easy to spot from a regular dump. The same applies to LFH blocks; they have a different type of metadata stored in the header, partially randomized as well. To dump these details, the !heap
–i command in the debugger does all the work to retrieve the metadata fields from a block, flagging checksum or free list inconsistencies as well if they exist. The command works for both the LFH and regular heap blocks. The total size of the blocks, the user requested size, the segment owning the block, as well as the header partial checksum are available in the output, as shown in the following sample. Because the randomization algorithm uses the heap granularity, the !heap –i command should be used only in the proper context of the heap containing the block. In the example, the heap handle is 0x001a0000. If the current heap context was different, the decoding of the header would be incorrect. To set the proper context, the same !heap –i command with the heap handle as an argument needs to be executed first.

作为堆块元数据随机化的一个影响,使用调试器简单地将一个块头部作为一个内存区域转储变得不是很有效。例如,从一个常规的转储中不容易辨认该块的大小以及是否处于忙碌状态。这同样适用于LFH块;它们在头部存储不同类型的元数据,只有部分进行了随机化。要转储这些细节,内核调试器命令  !heap –i  完成从一个块中检索元数据字段的所有工作,标志校验和或空闲列表(如果有的话)的不一致。该命令对于LFH和常规堆块都适用。下面例子中的输出展示了块的总大小,用户(应用)请求的大小,该块所属的段,以及头部校验和等信息。由于随机化算法使用的堆粒度,命令  !heap –i  应该仅用在包含特定块的适当堆上下文中。例子中的堆句柄为 0x001a0000。假设当前堆的上下文不同,解码出的头部信息将是错误的。要设置适当的上下文,需要首先执行同样使用堆句柄作为参数的  !heap –i  命令。(译注:为了便于直观理解,我在下面输出的重要行后面添加了C风格注释的说明)
[FONT="微软雅黑"][SIZE="4"][B]0:000> !heap -i 001a0000[/B]       //通过堆句柄指定特定堆的上下文

[B]Heap context set to the heap 0x001a0000 [/B]       

[B]0:000> !heap -i 1e2570 [/B]       //显示堆内特定块的信息

[B]Detailed information for block entry 001e2570[/B]

[B]Assumed heap : 0x001a0000 (Use !heap -i NewHeapHandle to change)[/B]

[B]Header content : 0x1570F4EC 0x0C0015BE (decoded : 0x07010006 0x0C00000D) [/B]       //该块的头部内容

[B]Owning segment : 0x001a0000 (offset 0) [/B]       //块所属的段,此例中所属段与所属堆相同

[B]Block flags : 0x1 (busy ) [/B]       //块标志,此例中该块处于忙碌状态

[B]Total block size : 0x6 units (0x30 bytes) [/B]       //块的总大小为48字节,包括6个单元,每单元8字节

[B]Requested size : 0x24 bytes (unused 0xc bytes) [/B]       //应用程序请求分配的大小为36字节,另外12字节未使用(36+12=48)

[B]Previous block size: 0xd units (0x68 bytes)  [/B]      //前一个块大小为104字节,包括13个单元,每单元8字节

[B]Block CRC : OK - 0x7[/B]        //本块的CRC(循环冗余校验)校验和正确

[B]Previous block : 0x001e2508 [/B]       //前一个块在本堆中的入口点

[B]Next block : 0x001e25a0[/B]        //下一个块在本堆中的入口点(验证:0x001e25a0 - 0x001e2570 = 0x30 = 48字节)[/SIZE][/FONT]

(译注:原文并没有从头到尾完整的介绍查看进程堆内存的步骤,下面给出使用命令行内核调试器KD.exe的实际调试过程截图,以供参考)
首先,进入livekd.exe所在路径,不带任何参数执行 livekd.exe,后者会转而创建一个 kd.exe 子进程;如果加上 -w参数,则livekd.exe 会创建一个windbg.exe 子进程:

2。执行  !process 0 0  命令,获取一份当前系统上所有活动进程的列表,第一个参数为进程的EPROCESS块(执行体进程块)的线性(虚拟地址),取值0时,将输出所有活动进程的信息;第二个参数设置输出的信息量,这是一个4位的16进制值,0x0信息最少;0xF信息最多:


3。我们从活动进程列表中选取一个内存占用较多的进程(因为可能有较多的进程堆使用率),下面的截图中,选取了 Adobe Acrobat Reader (功能之一为PDF阅读器)的主进程 AcroRd32.exe,将其EPROCESS块的地址作为  .process 命令的参数,这样会显式设置当前的进程上下文为AcroRd32.exe,注意,为了查看该进程堆,这个步骤是必须的:


4。从上面的输出可知,AcroRd32.exe 的 PEB(进程环境块)的虚拟地址为 7FFD4000,PEB是一个复杂的数据结构,其中存储有进程的堆句柄的信息;原文提到,堆调试命令  !heap  在首次使用时,需要后接一个进程堆句柄作为参数,以指定特定堆的上下文,因此,我们先查找到 PEB 中的堆句柄成员:

5。得到堆句柄值后,将其作为  !heap -i 命令的参数执行,设置堆上下文,然后将其作为  !heap -h 命令的参数执行,列出概要信息:

6。从下面节录的进程堆概要信息可知,第1个段中的第2个“堆块”的起始地址为00440588,如原文示例中所展示的,将这个块地址作为  !heap -i  命令的参数执行,就能查询特定堆块的元数据信息,另外,注意图中的前一块地址与下一块地址,它们与元数据中的相应字段匹配:


堆管理器使用 _HEAP 结构,来记录和维护堆的管理信息,这个结构位于每个进程堆的起始处,因此又称为堆的头部结构;尽管  !heap -i  命令能够显示部分的堆头部结构信息,更常规的做法是使用  dt ntdll!_HEAP  命令,后接堆的句柄或基址。这样可以列出完整的成员,如果加上 -r 开关,则可以遍历并展开所有嵌套的结构:

通过 KD.exe 的实际测试表明,启动 AcroRd32.exe 打开4个PDF文档时,其分配的堆约占用132MB内存;打开1个PDF文档时,其分配的堆约占用33MB内存,如最后这张图所示:

上面内容仅作为抛砖引玉,各位可以通过调试器命令和帮助文档,自行挖掘更丰富的信息。

Heap Debugging Features
The heap manager leverages the 8 bytes used to store internal metadata as a consistency checkpoint, which makes potential heap usage errors more obvious, and also includes several features to help detect bugs by using the following heap functions:
Enable tail checking    The end of each block carries a signature that is checked when the block is released. If a buffer overrun destroyed the signature entirely or partially, the heap will report this error.
Enable free checking    A free block is filled with a pattern that is checked at various points when the heap manager needs to access the block (such as at removal from the free list to satisfy an allocate request). If the process continued to write to the block after freeing it, the heap manager will detect changes in the pattern and the error will be reported.
Parameter checking    This function consists of extensive checking of the parameters passed to the heap functions.
Heap validation    The entire heap is validated at each heap call.
Heap tagging and stack traces support    This function supports specifying tags for allocation and/or captures user-mode stack traces for the heap calls to help narrow the possible causes of a heap error.

堆的调试功能
前文提到,堆管理器支持的最小分配粒度(堆块大小)在32位系统上为8字节;堆管理器利用这个存储内部元数据的大小,作为执行内存一致性检查的最小单元,这样细粒度的检查可以使得潜在的堆使用错误更加显而易见,并且包含多个有助于检测缺陷的功能,这可以通过使用下列堆函数实现:
启用尾端检查   每个堆块的结束部分都携带了一个签名,在释放时会进行检查。如果一个缓冲区溢出破坏了该签名的全部或部分,堆就会报告这个错误。
启用空闲检查   一个空闲块被填充了一种模式,当堆管理器需要访问该块时(例如从空闲列表中移除,以满足一个分配请求时),这种模式会在各个点上进行检查。如果进程在该块被释放后,仍持续向其中写入数据,堆管理器将检测到模式的改变并且回报一个错误。
参数检查   这个函数包括对传递给堆函数的参数进行的广泛校验。
堆验证   每次堆调用时都将验证整个堆。
堆标记与栈回溯支持   这个函数支持为堆分配指定标签,以及/或为堆调用捕获的用户模式栈回溯指定标签,这有助于缩小一个堆错误的可能原因。

The first three options are enabled by default if the loader detects that a process is started under the control of a debugger. (A debugger can override this behavior and turn off these features.) The heap debugging features can be specified for an executable image by setting various debugging flags in the image header using the Gflags tool. (See the section “Windows Global Flags” in Chapter
3 in Part 1.) Or, heap debugging options can be enabled using the !heap command in the standard Windows debuggers. (See the debugger help for more information.)

如果操作系统的应用程序加载器检测到一个进程在一个调试器的控制下启动,那么前3个选项默认是启用的。(调试器可以覆盖此行为并且关闭这些功能。)通过使用Gflags工具,在PE文件头中设置各种调试标志,就能为可执行文件映像指定堆的调试功能。(参考本书上册第3章“Windows Global Flags”一节。)或者,使用标准Windows调试器的  !heap  命令,也可以启用堆调试选项。(更多信息请参考调试器的帮助文档或命令)
Enabling heap debugging options affects all heaps in the process. Also, if any of the heap debugging options are enabled, the LFH will be disabled automatically and the core heap will be used (with the required debugging options enabled). The LFH is also not used for heaps that are not expandable (because of the extra overhead added to the existing heap structures) or for heaps that do not allow serialization.
启用堆调试功能将影响进程中的所有堆。此外,如果启用了(前述5项中的)任一堆调试选项,LFH将自动被禁用,并且将使用核心堆层(启用所需的调试选项时)。LFH也无法用于不可扩展的堆(因为LFH需要向现有堆结构中增加额外的开销),或者不允许序列化的堆。
Pageheap
Because the tail and free checking options described in the preceding sections might be discovering corruptions that occurred well before the problem was detected, an additional heap debugging capability, called pageheap, is provided that directs all or part of the heap calls to a different heap manager. Pageheap is enabled using the Gflags tool (which is part of the Debugging Tools for Windows). When enabled, the heap manager places allocations at the end of pages and reserves the immediately following page. Since reserved pages are not accessible, if a buffer overrun occurs it will cause an access violation, making it easier to detect the offending code. Optionally, pageheap allows placing the blocks at the beginning of the pages, with the preceding page reserved, to detect buffer underrun problems. (This is a rare occurrence.) The pageheap also can protect freed pages against any access to detect references to heap blocks after they have been freed.

页堆
由于前一节描述的尾端与空闲检查选项也许会在检测到问题前就发现产生的(堆块)损坏,因此提供了一个叫做pageheap(页堆)的额外堆调试能力,把所有或部分堆调用送往一个不同的堆管理器。使用Gflags工具能够启用页堆(Gflags工具位于Debugging Tools for Windows套件中,该套件还包括了图形界面内核调试器WinDbg.exe与命令行内核调试器KD.exe,套件可从微软主页下载)当启用页堆时,堆管理器将分配的内存放置于页面的结尾处,并且保留紧随其后的页面。由于这种保留页不可访问,如果发生缓冲区溢出将导致一个非法访问,使其更容易检测出问题代码。
另外,页堆还允许将堆块放置于页面的起始处,并保留之前的页面,这可用于检测缓冲区不足的问题。(这种情况很少见。)页堆还可以保护释放的页面,在堆块占用的内存被释放后,阻止任何试图探测其引用内存的访问。

Note that using the pageheap can result in running out of address space because of the significant overhead added for small allocations. Also, performance can suffer as a result of the increase of references to demand zero pages, loss of locality, and additional overhead caused by frequent calls to validate heap structures. A process can reduce the impact by specifying that the pageheap be used
only for blocks of certain sizes, address ranges, and/or originating DLLs.
For more information on pageheap, see the Debugging Tools for Windows Help file.

注意,由于为小范围内存分配增加了显著的开销(译注:小范围内存分配的定义请参考“堆管理器”部分开始的一段),使用页堆可能导致耗尽地址空间。此外,由于引用的“零需求页面”增加,局部性的损失,以及为了验证堆结构进行的频繁函数调用导致的额外开销。。。所有这些都会影响到性能。进程可以通过指定将页堆仅用于特定大小的块,地址范围,以及/或始发的DLL文件,来减轻对性能的影响。有关页堆的更多信息,请参考Debugging Tools for Windows套件中的帮助文档。
Fault Tolerant Heap
Corruption of heap metadata has been identified by Microsoft as one of the most common causes of application failures. Windows includes a feature called the fault tolerant heap, or FTH, in an attempt to mitigate these problems and to provide better problem-solving resources to application developers.
The fault tolerant heap is implemented in two primary components: the detection component, or FTH server, and the mitigation component, or FTH client.

容错堆
堆元数据的损坏已由微软确定为(导致)应用程序故障最常见的原因之一。Windows 包含一个叫做容错堆的特性,或 FTH,试图减轻这些问题,并且向应用程序开发人员提供更好的解决问题的资源。容错堆被实现为2个主要组件:检测(detection)组件,或 FTH 服务器;以及缓解(mitigation)组件,或 FTH 客户端。

The detection component is a DLL, Fthsvc.dll, that is loaded by the Windows Security Center service (Wscsvc.dll, which in turn runs in one of the shared service processes under the local service account). It is notified of application crashes by the Windows Error Reporting service.
检测组件被实现为 DLL 的形式,即 Fthsvc.dll,它由 Windows Security Center service 服务(Windows安全中心服务,即 Wscsvc.dll,它又运行在以本地服务帐户启动的一个共享服务进程中)加载。
由 Windows Error Reporting service (Windows 错误报告服务)向检测组件通知应用程序的崩溃消息。
(译注:通过在自己的机器上分析得知,系统中一个正常的 svchost.exe 进程之一会加载 Wscsvc.dll,将其作为Windows安全中心服务运行,它具体被实现为该进程内部的2个线程;而宿主进程,即这个 svchost.exe,则以 NT AUTHORITY\LOCAL SERVICE 帐户运行。基本与原文中描述相符,下图验证:)


When an application crashes in Ntdll.dll, with an error status indicating either an access violation or a heap corruption exception, if it is not already on the FTH service’s list of “watched” applications,
the service creates a “ticket” for the application to hold the FTH data. If the application subsequently crashes more than four times in an hour, the FTH service configures the application to use the FTH client in the future.

当一个应用程序在 Ntdll.dll 中崩溃,并且其错误状态指出要么为非法访问异常,要么为堆损坏异常时,假设该应用尚未在 FTH 服务的“监视的”(watched)应用程序列表中,则 FTH 服务创建一个该应用的“凭证”(ticket),用来保存 FTH 相关的数据。假设该应用在一个小时内的后续崩溃次数超过4次, FTH 服务会把该应用配置成在未来使用 FTH 客户端。
The FTH client is an application compatibility shim. This mechanism has been used since Windows XP to allow applications that depend on particular behavior of older Windows systems to run on later systems. In this case, the shim mechanism intercepts the calls to the heap routines and redirects them to its own code. The FTH code implements a number of “mitigations” that attempt to allow the application to survive despite various heap-related errors.
FTH 客户端是一个应用程序兼容性垫铺层(shim)。从 Windows XP 以来就使用此机制,它允许依赖于旧版 Windows 系统特定行为的应用程序能够在较新版本的系统上运行。在这种情况下,垫铺层机制拦截对堆例程的调用,并且将其重定向到垫铺层自身的代码中。该 FTH 代码实现了一系列的“缓解措施”(mitigations),尝试允许该应用程序存活下来,尽管该应用导致各种堆相关的错误。
For example, to protect against small buffer overrun errors, the FTH adds 8 bytes of padding and an FTH reserved area to each allocation. To address a common scenario in which a block of heap is accessed after it is freed, HeapFree calls are implemented only after a delay: ”freed” blocks are put on a list, and only freed when the total size of the blocks on the list exceeds 4 MB. Attempts to free regions that are not actually part of the heap, or not part of the heap identified by the heap handle argument to HeapFree, are simply ignored. In addition, no blocks are actually freed once exit or RtlExitUserProcess has been called.
举例来讲,为了防范小型的缓冲区溢出错误,FTH 向每个分配单元中添加了8字节的填充,以及一个 FTH 保留区域。为了解决这一常见的错误场景——访问一个被释放后的堆块——HeapFree() 调用仅在这一个延迟后才被实施:“释放的”块被置入一个列表中,并且仅当被置入列表中的块总大小超过4MB时,才实际释放。尝试释放实际上不属于该堆一部分的区域;或者在调用 HeapFree() 时,尝试释放不属于由堆句柄参数标识的堆一部分的区域,这些操作请求都将简单地被忽略。此外,一旦 exit() 或 RtlExitUserProcess() 被调用,就不再实际释放任何块。
The FTH server continues to monitor the failure rate of the application after the mitigations have been installed. If the failure rate does not improve, the mitigations are removed.
在安装了缓解措施(即 FTH 客户端)以后,FTH 服务器会持续监视应用程序的故障率。如果故障率没有得到改善,缓解措施将被移除。
The activity of the fault tolerant heap can be observed in the Event Viewer. Type eventvwr.msc at a Run prompt, and then navigate in the left pane to Event Viewer, Applications And Services Logs, Microsoft, Windows, Fault-Tolerant-Heap. Click on the Operational log. It may be disabled completely in the registry: in the key HKLM\Software\Microsoft\FTH, set the value Enabled to 0.
可以在事件查看器(Event Viewer)中,观察容错堆的活动情况。在“运行”提示符下输入 eventvwr.msc 并回车,然后依序导航到事件查看器的左侧窗格 -> 应用程序和服务日志 ->Microsoft -> Windows -> Fault-Tolerant-Heap。单击 Operational 日志即可查看与 FTH (或容错堆)服务的生命周期相关的事件记录。可以在注册表中完全禁用 FTH:键路径为 HKLM\Software\Microsoft\FTH,然后将其 Enabled 键值的数据设置为0即可。
The FTH does not normally operate on services, only applications, and it is disabled on Windows server systems for performance reasons. A system administrator can manually apply the shim to an application or service executable by using the Application Compatibility Toolkit.
FTH 通常不对服务进行操作,它仅管理应用程序(译注:或许是由于应用的故障率比系统服务要频繁),并且在 Windows 的服务器版本上,出于性能原因,FTH 是被禁用的。系统管理员通过使用 Application Compatibility Toolkit,可以手动将垫铺层应用至一个应用程序或服务的可执行文件上。

Virtual Address Space Layouts
虚拟地址空间布局
This section describes the components in the user and system address space, followed by the specific layouts on 32-bit and 64-bit systems. This information helps you to understand the limits on process and system virtual memory on both platforms.
本节描述了用户和系统地址空间中的组件,其次是32位系统与64位系统上的特定布局。这些信息有助于你理解在这2个平台上的进程和系统虚拟内存限制。
Three main types of data are mapped into the virtual address space in Windows: per-process private code and data, sessionwide code and data, and systemwide code and data.
三种主要类型的数据被映射到 Windows 中的虚拟地址空间:每进程的私有代码和数据,会话范围的代码和数据(译注:要理解会话这个概念,请参考本书上册第一章的相关内容),以及系统范围的代码和数据。
As explained in Chapter 1 in Part 1, each process has a private address space that cannot be accessed by other processes. That is, a virtual address is always evaluated in the context of the current process and cannot refer to an address defined by any other process. Threads within the process can therefore never access virtual addresses outside this private address space.
Even shared memory is not an exception to this rule, because shared memory regions are mapped into each participating process, and so are accessed by each process using per-process addresses. Similarly, the cross-process memory functions (ReadProcessMemory and WriteProcessMemory) operate by running kernel-mode code in the context of the target process.

正如本书上册第一章所讲解的,每个进程都拥有一个不可被其它进程访问的私有地址空间。这就是说,一个虚拟地址总是在当前进程的上下文中计算,并且无法引用一个由任何其它进程定义的地址。
因此,进程内的线程就绝不能访问在这个私有地址空间之外的虚拟地址。就这个规则而言,甚至共享内存也不例外,因为共享的内存区域被映射到每个参与的进程,而每个进程将使用每进程的地址来访问这些区域。类似地,跨进程内存函数(ReadProcessMemory() 和WriteProcessMemory())通过在目标进程的上下文中运行内核模式代码来运作。

The information that describes the process virtual address space, called page tables, is described in the section on address translation. Each process has its own set of page tables. They are stored in kernel-mode-only accessible pages so that user-mode threads in a process cannot modify their own address space layout.
描述进程虚拟地址空间的信息叫做页表,本节的地址翻译部分将对其进行解释。每个进程拥有自己的一组页表。它们被存储在仅内核模式下能够访问的页面中,这样,一个进程内的用户模式线程就无法修改它们自身的地址空间布局。
Session space contains information that is common to each session. (For a description of sessions, see Chapter 2 in Part 1.) A session consists of the processes and other system objects (such as the window station, desktops, and windows) that represent a single user’s logon session. Each session has a session-specific paged pool area used by the kernel-mode portion of
the Windows subsystem (Win32k.sys) to allocate session-private GUI data structures. In addition, each session has its own copy of the Windows subsystem process (Csrss.exe) and logon process (Winlogon.exe). The session manager process (Smss.exe) is responsible for creating new sessions, which includes loading a sessionprivate copy of Win32k.sys, creating the session-private object manager namespace, and creating the session-specific instances of the Csrss and Winlogon processes. To virtualize sessions, all sessionwide data structures are mapped into a region of system space called session space. When a process is created, this range of addresses is mapped to the pages associated with the session that the process belongs to.

会话空间包含了每个会话共用的信息。一个会话由多个进程以及其它系统对象(例如窗口站,桌面,以及窗口)构成,其代表单个用户的登录会话。每个会话有一个与该会话相关(专用)的可分页池区域,Windows 子系统的内核模式部分(Win32k.sys)使用该区域来分配会话私有的 GUI(图形用户界面) 数据结构。此外,每个会话都有自己的一份 Windows 子系统进程(Csrss.exe)和登录进程(Winlogon.exe)的副本。会话管理器进程(Smss.exe)负责创建新会话,这包括加载一个会话私有的 Win32k.sys 副本,创建会话私有的对象管理器名字空间,以及创建会话相关的 Csrss 和 Winlogon 进程实例。为了虚拟化会话,所有会话范围内的数据结构都被映射到一个叫做会话空间的系统空间区域。当一个进程被创建时,这个地址范围被映射到与该进程所属会话相关联的页面中。
Finally, system space contains global operating system code and data structures visible by kernel mode code regardless of which process is currently executing. System space consists of the following components:
■ System code                         Contains the operating system image, HAL, and device drivers used to boot the system.
■ Nonpaged pool                         Nonpageable system memory heap.
■ Paged pool                         Pageable system memory heap.
■ System cache                         Virtual address space used to map files open in the system cache. (See Chapter 11 for detailed information.)
■ System page table entries (PTEs)         Pool of system PTEs used to map system pages such as I/O space, kernel stacks, and memory descriptor lists. You can see how many system PTEs are available by examining the value of the Memory: Free System Page Table Entries counter in Performance Monitor.
■ System working set lists                 The working set list data structures that describe the three system working sets (the system cache working set, the paged pool working set, and the system PTEs working set).
■ System mapped views                 Used to map Win32k.sys, the loadable kernel-mode part of the Windows subsystem, as well as kernel-mode graphics drivers it uses. (See Chapter 2 in Part 1 for more information on Win32k.sys.)
■ Hyperspace                         A special region used to map the process working set list and other per-process data that doesn’t need to be accessible in arbitrary process                
context. Hyperspace is also used to temporarily map physical pages into the system space. One example of this is invalidating page table entries in page tables of processes other than the current one (such as when a page is removed from the standby list).
■ Crash dump information                 Reserved to record information about the state of a system crash.
■ HAL usage                         System memory reserved for HAL-specific structures.
Now that we've described the basic components of the virtual address space in Windows, let's examine the specific layout on the x86, IA64, and x64 platforms.

最后,系统空间包含了全局的操作系统代码和数据结构,无论当前正在执行哪个进程,它们都是内核模式代码可见的。系统空间由系列组件构成:
■ 系统代码                        包括用来启动系统的操作系统映像,HAL,以及设备驱动程序。
■ 非分页池                        不可分页的系统内存堆。
■ 分页池                        可分页的系统内存堆。
■ 系统缓存                        用来映射在系统缓存中打开的文件的虚拟地址空间。(更多细节请参考第11章)
■ 系统页表条目(PTEs)        用于映射诸如 I/O 空间,内核栈,以及内存描述符列表等系统页面的系统 PTE 池。你可以通过在性能监视器中查看“Memory: Free System Page Table Entries”这个计数器的值,来得知当前可用的系统 PTE 数量。
■ 系统工作集列表                该工作集列表数据结构描述了3类系统工作集(系统缓存工作集,可分页池工作集,还有系统 PTE 工作集)
■ 系统映射视图                用于映射 Win32k.sys,Windows 子系统的可加载内核模式部分,以及它使用的内核模式图形设备(显卡)驱动程序。
■ 超空间                        一个用于映射进程工作集列表和其它的每进程数据(它们不需要在任意进程的上下文中被访问)的特殊区域。超空间也用于临时将物理页面映射到系统空间。其中一个例子就是,除了进程页表中的无效页表条目外,引用的有效页面(例如,当一个页面从备用[standby]列表中被移除时)。
■ 崩溃转储信息                保留来记录与系统崩溃状态有关的信息。
■ HAL 使用情形                保留给 HAL 相关数据结构的系统内存。
到目前为止,我们已经描述了 Windows 中,虚拟地址空间的基础组件,下面我们将考察 x86,IA64,以及 x64 平台上的特定布局。


x86 Address Space Layouts

x86 地址空间布局
By default, each user process on 32-bit versions of Windows has a 2-GB private address space; the operating system takes the remaining 2 GB. However, the system can be configured with the increaseuserva BCD boot option to permit user address spaces up to 3 GB. Two possible address space layouts are shown in Figure 10-8
默认情况下,每个 32位版本 Windows 上的用户进程拥有一个 2GB 私有地址空间;操作系统占用剩余的 2GB。然而,系统可以被配置成以 increaseuserva BCD 启动选项来启动,这样就能够允许最多 3GB 的用户地址空间。下图 10-8 展示了2个可能的地址空间布局:(译注:注意下图中的内存地址是自顶向下增长的)


(译注:上图中有一个错误的地方。在其左侧地址空间布局中,存在一个用户与内核模式都不可访问的 64KB 区域,它的范围为 0x7FFF0000~ 0x7FFFFFFF,而图中给出的起始地址 0x7FFFF000 是错误的;另外,从 0x80000000 开始,即为内核空间。我们可以把这个 64KB,16进制为 FFFF 的“断层”想象为用户与内核空间的分水岭)
(译注:为了对图 10-8 的准确性加以验证,并从多方信息源加深对 x86 内核空间布局的理解,这里使用来自 CodeMachine 站点开发的 WinDbg 扩展 DLL—— CMKD.dll ,下载此动态链接库文件后,将其存放在 kd.exe/Windbg.exe
的目录,就可以通过 !cmkd.kvas 调试器扩展命令,列出 32 位 Windows 的内核空间布局信息,包括起始与结束地址,以 MB 计算的大小,数量,类型等等。因此这是一个强大的内核调试器扩展,推荐使用。各位可以从链接 http://www.codemachine.com/tools_x86/cmkd.dll  处下载。下面的截图是在我的 32 位 Windows 7 旗舰版上输出的内核空间布局:)



The ability for a 32-bit process to grow beyond 2 GB was added to accommodate the need for 32-bit applications to keep more data in memory than could be done with a 2-GB address space. Of course, 64-bit systems provide a much larger address space.

使用 2 GB 地址空间配置时,32位应用程序在其内存中能够保留的数据有限;为了适应对存储更多数据的需求,于是就添加了一个 32 位进程能够增长到超越 2 GB 地址空间的能力。当然,64 位系统会提供一个比这大得多的地址空间。
For a process to grow beyond 2 GB of address space, the image file must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header. Otherwise, Windows reserves the additional address space for that process so that the application won't see virtual addresses greater than 0x7FFFFFFF. Access to the additional virtual memory is opt-in because some
applications have assumed that they'd be given at most 2 GB of the address space. Since the high bit of a pointer referencing an address below 2 GB is always zero, these applications would use the high bit in their pointers as a flag for their own data, clearing it, of course, before referencing the data. If they ran with a 3-GB address space, they would inadvertently truncate pointers that have values greater than 2 GB, causing program errors, including possible data corruption. You set this flag by specifying the linker flag /LARGEADDRESSAWARE when building the executable. This flag has no effect when running the application on a system with a 2-GB user address space.

为了让一个进程能够增长到超过 2 GB 的地址空间,在其映像文件的映像头部(译注:通常是 PE 文件头)中必须设置了 IMAGE_FILE_LARGE_ADDRESS_AWARE 标志。否则,对于该进程,Windows 将“留住”( reserves)额外的地址空间,以至于超过 0x7FFFFFFF 以上的虚拟地址对于该应用程序而言是不可见的。由于某些应用程序假设了它们仅被给予最多 2 GB 的地址空间,因此对额外虚拟内存的访问完全是可选(opt-in)的。既然一个引用低于 2 GB 地址的指针其最高位始终为0,这些应用程序会将它们指针中的最高位用作一个标记,来标识其自身数据的边界;当然,在引用这些应用数据前,必须将最高位“清位”( clearing it)。(译注:由于一个低于 2 GB 的地址,如 0x7FFFFFFF,其二进制形式为 01111111111111111111111111111111,因此其最高位始终为0;而 2 GB 以上的任何一个地址,例如 0x80000000,其二进制形式为 10000000000000000000000000000000,其最高位始终为1)
就上述场景而言,如果这些应用数据(和代码)运行在一个 3 GB 的地址空间中,它们可能会无意间截断其值大于 2 GB 的指针,导致程序出错,包括可能的数据损坏。
你可以通过在构建相应可执行文件时,指定 /LARGEADDRESSAWARE 链接器标志,配置应用程序使用 3 GB 的地址空间。但是,当在一个配置成使用 2 GB 用户地址空间的系统上运行该应用程序时,此链接器标志将不起作用。(译注:也就是说,必须同时启用  increaseuserva BCD 启动选项,以及指定 /LARGEADDRESSAWARE 链接器标志;关于启用前者,请参考第一部分译文,这里不再赘述 )

Several system images are marked as large address space aware so that they can take advantage of systems running with large process address spaces. These include:
■ Lsass.exe         The Local Security Authority Subsystem
■ Inetinfo.exe         Internet Information Server
■ Chkdsk.exe         The Check Disk utility
■ Smss.exe         The Session Manager
■ Dllhst3g.exe         A special version of Dllhost.exe (for COM+ applications)
■ Dispdiag.exe         The display diagnostic dump utility
■ Esentutl.exe         The Active Directory Database Utility tool

下面这些系统映像被标记为察觉到大地址空间,以便它们能够充分利用以大进程地址空间运行的系统优势:
■ Lsass.exe         本地安全授权子系统
■ Inetinfo.exe         英特网信息服务器
■ Chkdsk.exe        Check Disk 实用工具
■ Smss.exe        会话管理器
■ Dllhst3g.exe         Dllhost.exe 的特别版本(用于 COM+ 应用程序)
■ Dispdiag.exe        显示诊断转储实用工具
■ Esentutl.exe         活动目录数据库实用工具

EXPERIMENT: Checking If an Application Is Large Address Aware
实验:检查应用程序是否启用了大地址空间
You can use the Dumpbin utility from the Windows SDK to check other executables to see if they support large address spaces. Use the /HEADERS flag to display the results. Here's a sample output of Dumpbin on the Session Manager:
你可以使用 Windows SDK 中的 Dumpbin 实用工具,检查其它的可执行文件是否支持大地址空间。使用 /HEADERS 标记能够显示相关的结果。下面这个输出样本是 Dumpbin 对会话管理器进程(smss.exe)的检测结果:
[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]C:\Program Files\Microsoft SDKs\Windows\v7.1>dumpbin /headers c:\windows\system32\smss.exe

Microsoft (R) COFF/PE Dumper Version 10.00.40219.01

Copyright (C) Microsoft Corporation. All rights reserved.

Dump of file c:\windows\system32\smss.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES

8664 machine (x64)

5 number of sections

4A5BC116 time date stamp Mon Jul 13 16:19:50 2009

0 file pointer to symbol table

0 number of symbols

F0 size of optional header

22 characteristics

Executable

Application can handle large (>2GB) addresses[/COLOR][/SIZE][/FONT]

(译注:由于当前发行的  Windows SDK 中仅包含 64 位的 dumpbin.exe ,因此若要检测 32 位可执行文件,可以通过 PEView 或其它类似的 PE 文件格式查看工具来完成,下面这张截图是使用 PEView 检查 smss.exe 是否支持大地址空间的结果:)


Finally, because memory allocations using VirtualAlloc, VirtualAllocEx, and VirtualAllocExNuma start with low virtual addresses and grow higher by default, unless a process allocates a lot of virtual memory or it has a very fragmented virtual address space, it will never get back very high virtual addresses. Therefore, for testing purposes, you can force memory allocations to
start from high addresses by using the MEM_TOP_DOWN flag or by adding a DWORD registry value, HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference, and setting it to 0x100000.

最后,由于使用 VirtualAlloc()VirtualAllocEx(),以及 VirtualAllocExNuma() 的内存分配操作,默认会从虚拟内存的低址段开始分配,然后往高址段增长,所以除非一个进程分配了大量的虚拟内存,或者它有一个非常零碎(分散)的虚拟地址空间,否则它永远都不会回到非常高的虚拟地址上来。故而,出于测试目的,你可以通过使用 MEM_TOP_DOWN 标志,强制内存分配操作从高址段开始并且往低址段增长;或者添加一个名为 AllocationPreference 的 DWORD 类型注册表键值,路径为:HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\ ,然后将该键值的数据设置为 0x100000。
Figure 10-9 shows two screen shots of the TestLimit utility (shown in previous experiments) leaking memory on a 32-bit Windows machine booted with and without the increaseuserva option set to 3 GB.
下图 10-9 中的2个屏幕截图展示了在一个 32 位 Windows 机器上分别开启 increaseuserva (3 GB 用户空间)和关闭该选项然后启动,使用 TestLimit 实用工具(在前面的实验中用过)执行内存泄露操作的结果。
Note that in the second screen shot, TestLimit was able to leak almost 3 GB, as expected. This is only possible because TestLimit was linked with /LARGEADDRESSAWARE. Had it not been, the results would have been essentially the same as on the system booted without increaseuserva.
注意第二个屏幕截图,其输出表明 TestLimit 能够泄露 3 GB,正如预期的一样。这还需要 TestLimit 应用程序以 /LARGEADDRESSAWARE 链接器标志来构建才行。如果没有指定该标记,那么测试的结果将基本上与关闭 increaseuserva 启动选项时的相同。(即第一个截图)  

x86 System Address Space Layout
x86 系统地址空间布局
The 32-bit versions of Windows implement a dynamic system address space layout by using a virtual address allocator (we'll describe this functionality later in this section). There are still a few specifically reserved areas, as shown in Figure 10-8. However, many kernel-mode structures use dynamic address space allocation. These structures are therefore not necessarily
virtually contiguous with themselves. Each can easily exist in several disjointed pieces in various areas of system address space. The uses of system address space that are allocated in this way include:
■ Nonpaged pool
■ Special pool
■ Paged pool
■ System page table entries (PTEs)
■ System mapped views
■ File system cache
■ File system structures (metadata)
■ Session space

32 位版本的 Windows 通过使用一个虚拟地址分配器(本节稍后将讨论此功能)实现了一个动态的系统地址空间布局。如图 10-8 所示,系统地址空间中仍然有一些专门的保留区域。然而,许多内核模式数据结构使用动态地址空间来分配。因此这些数据结构自身不一定非得几乎连续相邻。每一个都能够轻易地在系统地址空间的各个区域中,存在一些不连续的碎片。被以这种分配方式来使用系统地址空间的包括:
■  非分页池
■  专用(Special)池(译注:关于专用/特殊池的简介请参考译文的“内核模式堆[系统内存池]”一节;详细讨论请参考本书下册第14章)
■  分页池
■  系统页表条目(PTEs)
■  系统映射视图
■  文件系统缓存
■  文件系统结构(元数据)
■  会话空间

x86 Session Space
x86 会话空间
For systems with multiple sessions, the code and data unique to each session are mapped into system address space but shared by the processes in that session. Figure 10-10 shows the general layout of session space.
对于有多个会话的系统而言,每个会话独有的代码与数据被映射到系统地址空间,但是被该会话中的所有进程共享。下图 10-10 展示了会话空间的总体布局:

The sizes of the components of session space, just like the rest of kernel system address space, are dynamically configured and resized by the memory manager on demand.
会话空间中各种组件的大小,就如同内核系统地址空间的其余部分一样,由内存管理器按需动态配置和重新调整其大小。
EXPERIMENT: Viewing Sessions
实验:查看会话
You can display which processes are members of which sessions by examining the session ID. This can be viewed with Task Manager, Process Explorer, or the kernel debugger. Using the kernel debugger, you can list the active sessions with the !session command as follows:
你可以通过检查会话 ID 来显示哪些进程是哪些会话的成员。可以使用任务管理器,进程浏览器,或者内核调试器来检查。如果使用内核调式器的 !session 命令,可以列出活动的会话。如下所示:
[FONT="微软雅黑"][SIZE="4"]lkd> !session

Sessions on machine: 3

Valid Sessions: 0 1 3

Current Session 1[/SIZE][/FONT]

Then you can set the active session using the !session –s command and display the address of the session data structures and the processes in that session with the !sprocess command:
接下来,你可以使用命令 !session –s 设置活动的会话,然后通过 !sprocess 命令,就可以显示该会话数据结构的地址,以及该会话中的进程们:
[FONT="微软雅黑"][SIZE="4"]lkd> !session -s 3

Sessions on machine: 3

Implicit process is now 84173500

Using session 3

lkd> !sprocess

Dumping Session 3

_MM_SESSION_SPACE 9a83c000

_MMSESSION 9a83cd00

PROCESS 84173500 SessionId: 3 Cid: 0d78 Peb: 7ffde000 ParentCid: 0e80

DirBase: 3ef53500 ObjectTable: 8588d820 HandleCount: 76.

Image: csrss.exe

PROCESS 841a6030 SessionId: 3 Cid: 0c6c Peb: 7ffdc000 ParentCid: 0e80

DirBase: 3ef53520 ObjectTable: 85897208 HandleCount: 94.

Image: winlogon.exe

PROCESS 841d9cf0 SessionId: 3 Cid: 0d38 Peb: 7ffd6000 ParentCid: 0c6c

DirBase: 3ef53540 ObjectTable: 8589d248 HandleCount: 165.

Image: LogonUI.exe[/SIZE][/FONT]

To view the details of the session, dump the MM_SESSION_SPACE structure using the dt command, as follows:
要查看特定会话的详细信息,可以使用 dt 命令,转储上面输出中, MM_SESSION_SPACE 结构的地址,如下所示:

[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]lkd> dt nt!_MM_SESSION_SPACE 9a83c000

+0x000 ReferenceCount : 0n3

+0x004 u : <unnamed-tag>

+0x008 SessionId : 3

+0x00c ProcessReferenceToSession : 0n4

+0x010 ProcessList : _LIST_ENTRY [ 0x841735e4 - 0x841d9dd4 ]

+0x018 LastProcessSwappedOutTime : _LARGE_INTEGER 0x0

+0x020 SessionPageDirectoryIndex : 0x31fa3

+0x024 NonPagablePages : 0x19

+0x028 CommittedPages : 0x867

+0x02c PagedPoolStart : 0x80000000 Void

+0x030 PagedPoolEnd : 0xffbfffff Void

+0x034 SessionObject : 0x854e2040 Void

+0x038 SessionObjectHandle : 0x8000020c Void

+0x03c ResidentProcessCount : 0n3

+0x040 SessionPoolAllocationFailures : [4] 0

+0x050 ImageList : _LIST_ENTRY [ 0x8519bef8 - 0x85296370 ]

+0x058 LocaleId : 0x409

+0x05c AttachCount : 0

+0x060 AttachGate : _KGATE

+0x070 WsListEntry : _LIST_ENTRY [ 0x82772408 - 0x97044070 ]

+0x080 Lookaside : [25] _GENERAL_LOOKASIDE

...[/COLOR][/SIZE][/FONT]

EXPERIMENT: Viewing Session Space Utilization
实验:查看会话空间的使用率
You can view session space memory utilization with the !vm 4 command in the kernel debugger. For example, the following output was taken from a 32-bit Windows client system with the default two sessions created at system startup:
你可以通过内核调式器中的 !vm 4 命令,查看会话空间的内存使用情况。例如,下面的输出来自于一个 32 位的客户版 Windows 系统,其系统在启动时,默认会创建 2 个会话:(译注:通常就是会话 0 与 1)
[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]lkd> !vm 4

.

.

Terminal Server Memory Usage By Session:

Session ID 0 @ 9a8c7000:

Paged Pool Usage: 2372K

Commit Usage: 4832K

Session ID 1 @ 9a881000:

Paged Pool Usage: 14120K

Commit Usage: 16704K[/COLOR][/SIZE][/FONT]

System Page Table Entries
系统页表条目
System page table entries (PTEs) are used to dynamically map system pages such as I/O space, kernel stacks, and the mapping for memory descriptor lists. System PTEs aren't an infinite resource. On 32-bit Windows, the number of available system PTEs is such that the system can theoretically describe 2 GB of contiguous system virtual address space. On 64-bit Windows, system PTEs can describe up to 128 GB of contiguous virtual address space.
系统页表条目(PTEs)用于动态地映射系统页面,例如 I/O 空间,内核栈,以及映射内存描述符列表(MDLs)。系统页表条目并非是一种取之不尽,用之不竭的资源。在 32 位 Windows 系统上,可用的系统页表条目数量为:系统理论上能够描述的 2 GB 连续系统虚拟地址空间。在 64 位 Windows 系统上,系统页表条目能够描述最多 128 GB 的连续虚拟地址空间。
EXPERIMENT: Viewing System PTE Information
实验:查看系统页表条目相关信息
You can see how many system PTEs are available by examining the value of the Memory: Free System Page Table Entries counter in Performance Monitor or by using the !sysptes or !vm command in the debugger. You can also dump the _MI_SYSTEM_PTE_TYPE structure associated with the MiSystemPteInfo global variable. This will also show you how many PTE allocation
failures occurred on the system—a high count indicates a problem and possibly a system PTE leak.

你可以在性能监视器中,添加并查看 Memory: Free System Page Table Entries 计数器的值,来得知可用的系统页表条目数量,或者通过使用 !sysptes!vm 内核调试器命令。你也可以转储 _MI_SYSTEM_PTE_TYPE 结构,该结构是与 MiSystemPteInfo 这个全局变量相关联的。最后一种方法(转储)还可以向你显示出,系统上发生 PTE 分配失败的次数——如果这个值很大则表明一个问题,可能与系统 PTE 泄露有关。
[FONT="微软雅黑"][SIZE="4"]0: kd> !sysptes

System PTE Information

[COLOR="Red"]Total System Ptes 307168   //该值为10进制形式,可以对照性能监视器中的值来验证[/COLOR]

starting PTE: c0200000

free blocks: 32 total free: 3856 largest free block: 542

Kernel Stack PTE Information

Unable to get syspte index array - skipping bins

starting PTE: c0200000

free blocks: 165 total free: 1503 largest free block: 75

[COLOR="Red"]0: kd> ? nt!MiSystemPteInfo[/COLOR]

[COLOR="Red"]Evaluate expression: -2100014016 = 82d45440

0: kd> dt nt!_MI_SYSTEM_PTE_TYPE 82d45440[/COLOR]

nt!_MI_SYSTEM_PTE_TYPE

+0x000 Bitmap : _RTL_BITMAP

+0x008 Flags : 3

+0x00c Hint : 0x2271f

+0x010 BasePte : 0xc0200000 _MMPTE

[COLOR="Red"]+0x014 FailureCount : 0x82d45468 -> 0[/COLOR]

+0x018 Vm : 0x82d67300 _MMSUPPORT

+0x01c TotalSystemPtes : 0n7136

+0x020 TotalFreeSystemPtes : 0n4113

+0x024 CachedPteCount : 0n0

[COLOR="Red"]+0x028 PteFailures : 0[/COLOR]

+0x02c SpinLock : 0

+0x02c GlobalMutex : (null)[/SIZE][/FONT]

If you are seeing lots of system PTE failures, you can enable system PTE tracking by creating a new DWORD value in the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management key called TrackPtes and setting its value to 1. You can then use !sysptes 4 to show a list of allocators, as shown here:
如果你看到了大量的系统 PTE 分配失败,你可以通过创建一个新的,叫做 TrackPtes 的注册表键(DWORD 类型)来启用系统 PTE 跟踪,其路径为:HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management ,并将值设置为 1。然后你就可以使用 !sysptes 4 调式器命令,显示(系统 PTE)分配者的列表,如下所示:(译注:修改注册表后可能需要重启系统,一方面使得设置生效,然后再次运行内核调试器,另一方面是由于使用 Livekd.exe 只是通过转储快照来模拟某一时点的内核状态,而非真正的实时内核调试;真正的实时内核调试需要物理双机器环境通过 COM 串口上的 RS-232/CB9/DB9 线缆,或者 USB 2.0 调试线缆,或者 IEEE 1394 线缆,或者一个虚拟机和物理机通过命名管道连接)

[FONT="微软雅黑"][SIZE="4"][COLOR="Black"]lkd>!sysptes 4

0x1ca2 System PTEs allocated to mapping locked pages

VA(在系统PTE区域内分配的起始虚拟地址) 	MDL(内存描述符列表) 	PageCount(分配的大小/页面数) 	Caller/CallersCaller

ecbfdee8					f0ed0958 			2 				netbt!DispatchIoctls+0x56a/netbt!NbtDispatchDevCtrl+0xcd

f0a8d050 					f0ed0510 			1 				netbt!DispatchIoctls+0x64e/netbt!NbtDispatchDevCtrl+0xcd

ecef5000 					1 			20 				nt!MiFindContiguousMemory+0x63

ed447000 					0 			2 				Ntfs!NtfsInitializeVcb+0x30e/Ntfs!NtfsInitializeDevice+0x95

ee1ce000 					0 			2 				Ntfs!NtfsInitializeVcb+0x30e/Ntfs!NtfsInitializeDevice+0x95

[COLOR="Red"][B]ed9c4000 					1 			ca 				nt!MiFindContiguousMemory+0x63[/B][/COLOR]

eda8e000 					1 			ca 				nt!MiFindContiguousMemory+0x63

efb23d68 					f8067888 			2 				mrxsmb!BowserMapUsersBuffer+0x28

efac5af4 					f8b15b98 			2 				ndisuio!NdisuioRead+0x54/nt!NtReadFile+0x566

f0ac688c 					f848ff88 			1 				ndisuio!NdisuioRead+0x54/nt!NtReadFile+0x566

efac7b7c 					f82fc2a8 			2 				ndisuio!NdisuioRead+0x54/nt!NtReadFile+0x566

ee4d1000 					1 			38 				nt!MiFindContiguousMemory+0x63

[COLOR="Red"][B]f0676000 					1 			10 				hal!HalpGrowMapBuffers+0x134/hal!HalpAllocateAdapterEx[/B][/COLOR]

[/COLOR][/SIZE][/FONT]

(译注:从上面的输出可以观察到,系统 PTE 的分配者除了各种执行体/内核组件外,多数是一些加载到内核空间的设备驱动程序,其中有系统自带的,也有第三方软硬件供应商开发的;
在“调用者/调用者的调用者”一列中,感叹号前面的就是驱动程序文件名,其省略了.sys文件名后缀,这些驱动程序的二进制文件,绝大多数可以在 C:\Windows\System32\drivers 路径下找到;感叹号右侧的则是该驱动中,请求MiObtainSystemVa() 分配系统 PTE 的例程(包括直接或间接递归调用),以及偏移位置。从这些例程的名称可以看出,它们请求在系统 PTE 区域中分配内存的目的都是与映射视图,MDLs(内存描述符列表),适配器内存映射,驱动程序映像,内核栈, I/O 映射等相关的。例如,MiFindContiguousMemory() -> MmMapIoSpace() -> MiInsertIoSpaceMap -> ExAllocatePoolWithTag() -> MiAllocatePoolPages() -> MiAllocatePagedPoolPages() -> MiObtainSystemVa()
由此可知,无论是直接还是间接调用 MiObtainSystemVa() ,来在系统 PTE 区域分配内存,都会被跟踪记录下来(启用系统 PTE 分配者跟踪后)。
再如 hal!HalpAllocateAdapterEx ,从其名称就可以推测出是与适配器内存映射相关的例程。我们可以直接使用调试器命令 u 后接完整的例程偏移位置,来反汇编涉及分配操作地址处的机器代码,下面是在真实机器上的输出,注意,Caller/CallersCaller 一列被“ Callstack”取代。另外,毕竟是用 Livekd 单机模拟实时调试,下图结果不够准确或者容易让人迷惑,例如无法定位确切的 PTE 分配代码。因此,建议有物理双机环境的童鞋,可以自行验证看看)


64-Bit Address Space Layouts
64 位地址空间布局
The theoretical 64-bit virtual address space is 16 exabytes (18,446,744,073,709,551,616 bytes, or approximately 18.44 billion billion bytes). Unlike on x86 systems, where the default address space is divided in two parts (half for a process and half for the system), the 64-bit address is divided into a number of different size regions whose components match conceptually the
portions of user, system, and session space. The various sizes of these regions, listed in Table 10-8, represent current implementation limits that could easily be extended in future releases. Clearly, 64 bits provides a tremendous leap in terms of address space sizes.

理论上的 64 位虚拟地址空间大小为 16 EB(18,446,744,073,709,551,616 字节,或者大约 18.44 十亿十亿字节;1 EB 等于 10 的 18 次方字节)。与 x86 系统上的地址空间布局不一样(其默认地址空间被划分为两部分,一半用于一个进程;另一半用于系统)的是,64 位地址空间被划分成多个不同大小的区域,其组成与概念上的用户,系统,以及会话空间等部分相匹配。表 10-8 列出了这些区域的各种大小;需要指出,其中的数值仅仅代表当前实现中的限制,在后续的发布版中,这些值能够轻易地被扩大(译注:想象一下,16 EB 可以“容纳”多少个 GB,TB 以及 PB 单位)。很显然,“64 位”提供了
地址空间尺寸方面的一个巨大飞跃。(译注:以各位读到这里的水平而言,下表应该能轻松看懂,就不翻译了)


Also, on 64-bit Windows, another useful feature of having an image that is large address space aware is that while running on 64-bit Windows (under Wow64), such an image will actually receive all 4 GB of user address space available—after all, if the image can support 3-GB pointers, 4-GB pointers should not be any different, because unlike the switch from 2 GB to 3 GB,
there are no additional bits involved. Figure 10-11 shows TestLimit, running as a 32-bit application, reserving address space on a 64-bit Windows machine, followed by the 64-bit version of TestLimit leaking memory on the same machine.

此外,在 64 位 Windows 上,另一个有用的功能就是,假设一个标记了意识(aware)到大地址空间的映像文件在 64 位 Windows 上运行时(在  Wow64 下),那么该映像将真正获得全部可用的 4 GB 用户地址空间——毕竟,如果该映像能够支持 3 GB 的指针,那么 4 GB 的指针应该不会有任何差异,这是由于,与从 2 GB 切换到 3 GB 不同, 从 3 GB 切换到 4 GB 不会涉及额外的位。图 10-11 展示了作为一个 32 位应用程序运行的 TestLimit 程序在一个 64 位 Windows 机器上保留的地址空间大小;该图的下半部分则是 TestLimit 程序的 64 位版本在相同机器上泄露(leaking)的内存大小。
(译注:从下半部分输出可以看到,64位 TestLimit 程序能够泄露 8388548 MB ,即大约 8388 GB 的内存,这与表 10-8 中的 x64 系统上的进程地址空间限制——8192 GB 非常近似;而上半部分——32 位 TestLimit 在 64 位机器上至多能够泄露 4031 MB ,即 4 GB 内存——也与原文中的描述相符)


Note that these results depend on the two versions of TestLimit having been linked with the /LARGEADDRESSAWARE option. Had they not been, the results would have been about 2 GB for each. 64-bit applications linked without /LARGEADDRESSAWARE are constrained to the first 2 GB of the process virtual address space, just like 32-bit applications.
注意,这些输出的结果依赖于 TestLimit 程序的两个版本(32 位与 64 位)已经使用 /LARGEADDRESSAWARE 选项进行了链接。如果不是这样,每个版本的输出结果将约为 2 GB。没有使用 /LARGEADDRESSAWARE 链接器选项构建的 64 位应用程序被限制为仅能使用该进程虚拟地址空间的首个 2 GB 范围,这就与 32 位应用程序一样了。

注意:上传附件及图片大小不得大于30M。

⚠️ 版权声明:
本博客所有内容(含教程、源码、工具)仅供个人技术学习与研究交流使用,严禁商用、倒卖、二次分发及非法用途
未经作者书面授权,任何组织或个人不得转载、复制或用于其他平台,违者将追究相关责任。

0 0 0 举报
复制成功