reasons why malloc fails?

[prasath]

hi,

will someone point me the reasons why a call to malloc fails even if we have enough memory ?

[SpS]

will someone point me the reasons why a call to malloc fails even if we have enough memory ?

show your code which is failing and make sure you are not passing -1 in malloc.

[dubeyprateek]

Every process in Windows NT has one heap called the default heap. The Win32 subsystem uses the default heap for all global and local memory management functions, and the C run-time library uses the default heap for supporting malloc functions.

maaloc can fail as it cant allocate memory bigger than default heap. not all system memory is available to malloc.

[Ancient Dragon]

The in MS-Windows the default heap is not static -- the os will expand it as needed. But since malloc() takes an size_t integer as the parameter the largest amount of memory malloc can allocate at one time is the largest value that can be stored in the size_t integer (see limits.h). And yes, all the computer's available and unused memory can be used by malloc with the previously mentioned restriction.

malloc() normally fails today for one of two reason: (1) attempt to allocate more memory then is available, or (2) memory has been previously trashed (most common reason) such as buffer overflows and using uninitialized pointers (although there are a whole host of other causes).

[Salem]

> will someone point me the reasons why a call to malloc fails even if we have enough memory ?
My guess is you're still using crusty old TurboC which is limited to 640K no matter how many GB of memory you have on your pentium powered, XP hosted machine.

"Hello, yes I'd like to buy a Ferrari please"
"Certainly sir, will you be wanting to replace the engine with an elastic band?"

[glmohan2002]

The in MS-Windows the default heap is not static -- the os will expand it as needed. But since malloc() takes an size_t integer as the parameter the largest amount of memory malloc can allocate at one time is the largest value that can be stored in the size_t integer (see limits.h). And yes, all the computer's available and unused memory can be used by malloc with the previously mentioned restriction.

malloc() normally fails today for one of two reason: (1) attempt to allocate more memory then is available, or (2) memory has been previously trashed (most common reason) such as buffer overflows and using uninitialized pointers (although there are a whole host of other causes).

HI, I am also facing a same problem, " Malloc fails". Later I found the reason why malloc was failing. It was failing due to Buffer Overflow. But I am not able to understand why malloc fails due to Buffer Overflow. Can any one explain what exactly happens when there is a buffer over flow and how does it affect malloc. Thank you.

[Narue]

>Can any one explain what exactly happens when there
>is a buffer over flow and how does it affect malloc.
No, because that's an implementation detail. It depends on how malloc works for your system as well as what you're doing in your code. Post your code that fails (preferably a small example that still fails) and I can describe what you're doing wrong and speculate about what's happening behind the scenes.

[glmohan2002]

>Can any one explain what exactly happens when there
>is a buffer over flow and how does it affect malloc.
No, because that's an implementation detail. It depends on how malloc works for your system as well as what you're doing in your code. Post your code that fails (preferably a small example that still fails) and I can describe what you're doing wrong and speculate about what's happening behind the scenes.

The code is very simple. (I am giving the example)
One buffer is defined as "char buf[1001]" and "buf" is assigned with 1000 characters.
Later inside the code I am doing the following..

char *name, *temp;

name = (char *)malloc(500);

strcpy(name, buf);

temp = (char *)malloc(sizeof(char));

In this piece of code, name is allocated with 500 bytes and 1000 bytes are copied into this memory resulting in a buffer overflow. Due to this malloc at "temp = (char *)malloc(sizeof(char));" instruction is failing. I am trying to understand how this buffer overflow is causing a problem to malloc even though there is a sufficient memory is available.

[Narue]

>I am trying to understand how this buffer overflow is causing a problem
>to malloc even though there is a sufficient memory is available.
It's not about there being sufficient memory. If you've corrupted the memory manager's housekeeping data, it can't properly do its job. A buffer overflow can easily cause that kind of corruption.

[dubeyprateek]

Your heap is divided into segments you can have maximum 64 segments and each segment is divided into blocks. The blocks are the actuall memory you allocate.

The first heap block is an allocation of 8 bytes. This block is also a free block.
The second heap block that follows it is an allocation of 16 bytes. This block is not free.

_______________________________________________________________________________________________________________________
|<--------------------------------------------------- HEAP SEGMENT ---------------------------------------------------->
|<------------------- HEAP BLOCK -------------->|<------------------------------ HEAP BLOCK --------------------------->
|<------- header ------>|<----- user data ----->|<------- header ------>|<----------------- user data ----------------->
|s- -- p- -- s- f- u- t- d- d- d- d- d- d- d- d- s- -- p- -- s- f- u- t- d- d- d- d- d- d- d- d- d- d- d- d- d- d- d- d-
|02 00 03 00 03 00 00 00 88 01 07 00 00 fa 0d 0c 03 00 02 00 03 01 0c 00 08 37 e2 7c 48 5e 77 05 01 00 00 00 00 00 00 00
| 0002  0003  0003  0000  0188  0007  fa00  0c0d  0003  0002  0103  000c  3708  7ce2  5e48  0577  0001  0000  0000  0000
|   00030002    00000003    00070188    0c0dfa00    00020003    000c0103    7ce23708    05775e48    00000001    00000000
|
|00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17
|00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27
|<------- header ------>|<----- user data ----->|<------- header ------>|<----------------- user data ----------------->
|_______________________________________________________________________________________________________________________

So in a heap block, the first 8 bytes is the heap header. In my case 02 00 03 00 03 00 00 00 is the heap header of the first block
03 00 02 00 03 01 0c 00 is the heap header of the second heap block.

First four bytes are the size of a heap block Once you run your code you actually corrupt the next heap block. You just overwrite the size. This does not effect functioning of malloc but it corrups the memory, once you try to access that memory you expect a crash due to access denied error ..!!

[Ancient Dragon]

>>Your heap is divided into segments you can have maximum 64 segments <blabla>
That also is implementation dependent. I think that is how it works in ancient 16-bit MS-DOS but things have change just a tad since the 1980s. Memory is no longer segmented like that.

[dubeyprateek]

>>That also is implementation dependent. I think that is how it works in ancient 16-bit MS-DOS but things have change just a tad since the 1980s. Memory is no longer segmented like that.

I am not sure how it use to work in 16 bit DOS but I am sure this is how it works in Windows Vista ! :)

[Ancient Dragon]

I am not sure how it use to work in 16 bit DOS but I am sure this is how it works in Windows Vista ! :)

Link please. No version of Windows since NT 4.0 have had segmented memory. The segment registers do not contain segments -- they contain descriptors.

>>you can have maximum 64 segments
Then why can I allocate as much memory as I want at one time up to the limit of size_t and size of RAM + hard drive? The 32-bit version of Windows can address up to 4 gig of physical RAM.

[dubeyprateek]

Seems I could not present myself correctly...
Heap and Virtual memory are different
see this for heaps
http://msdn2.microsoft.com/en-us/library/aa366711(VS.85).aspx
This http://msdn2.microsoft.com/en-us/library/aa366599.aspx cluearly states why allocations grater in size do not fails.
Heap manager manages heaps it is a part og Windows memory manager.

You can see "Windows Internals 4th Edition" or you can use Windbg to see the implimentational details of heaps in Windows.

Heap Manager
Many applications allocate smaller blocks than the 64-KB minimum allocation granularity possible using page granularity functions such as VirtualAlloc. Allocating such a large area for relatively small allocations is not optimal from the memory usage and performance standpoint. To address this need, Windows provides a component called the heap manager, which manages allocations inside larger memory areas reserved using the page granularity memory allocation functions. The allocation granularity in the heap manager is relatively small: 8 bytes on 32-bit systems and 16 bytes on 64-bit systems. The heap manager has been designed to optimize memory usage and performance in the case of these smaller allocations.
The heap manager exists in two places: Ntdll.dll and Ntoskrnl.exe. The subsystem APIs (such as the Windows heap APIs) call the functions in Ntdll, and various executive components and device drivers call the functions in Ntoskrnl. Its native interfaces (prefixed with Rtl) are available only for use in internal Windows components or kernel mode device drivers. The documented Windows API interface to the heap (prefixed with Heap) are thin functions that call the native functions in Ntdll.dll. In addition, legacy APIs (prefixed with either Local or Global) are provided to support older Windows applications. The most common Windows heap functions are:
• HeapCreate or HeapDestroy—Creates or deletes, respectively, a heap. The initial reserved and committed size can be specified at creation.
• HeapAlloc—Allocates a heap block.
• HeapFree—Frees a block previously allocated with HeapAlloc.
• HeapReAlloc—Changes the size of an existing allocation (grows or shrinks an existing block).
• HeapLock or HeapUnlock—Controls mutual exclusion to the heap operations.
• HeapWalk—Enumerates the entries and regions in a heap.
Types of Heaps
Each process has at least one heap: the default process heap. The default heap is created at process startup and is never deleted during the process's lifetime. It defaults to 1 MB in size, but it can be bigger by specifying a starting size in the image file by using the /HEAP linker flag. This size is just the initial reserve, however—it will expand automatically as needed. (You can also specify the initial committed size in the image file.)
The default heap can be explicitly used by a program or implicitly used by some Windows internal functions. An application can query the default process heap by making a call to the Windows function GetProcessHeap. Processes can also create additional private heaps with the HeapCreate function. When a process no longer needs a private heap, it can recover the virtual address space by calling HeapDestroy. An array with all heaps is maintained in each process, and a thread can query them with the Windows function GetProcessHeaps.
A heap can manage allocations either in large memory regions reserved from the memory manager via VirtualAlloc or from memory mapped file objects mapped in the process address space. The latter approach is rarely used in practice, but it's suitable for scenarios where the content of the blocks need to be shared between two processes or between a kernel mode and a user mode component. If a heap is built on top of a memory mapped file region, certain constraints apply with respect to the component that can call heap functions. First, the internal heap structures use pointers, and therefore do not allow relocation to different addresses. Second, the synchronization across multiple processes or between a kernel component and a user process is not supported by the heap functions. Also, in the case of a shared heap between user and kernel mode, the user mode mapping should be read-only to prevent user mode code from corrupting the heap internal structures, which would result in a system crash.

[dubeyprateek]

PAE does not have to do anything with Heaps.

[Ancient Dragon]

PAE does not have to do anything with Heaps.

Only in that the heap can not access memory beyond the 4 gig boundry.

[dubeyprateek]

>>Only in that the heap can not access memory beyond the 4 gig boundry.

In fact heap can never access memory beyond 4 GB in 32 bit Windows Operating System.

I am sorry for not making myself clear. Virtual memory and heap are different. Heap is a part of virtual memory with low allocation granuality. It is all different set of APIs are given to access it, cretate it and distroy it.
PAE inclreases limit of Virtual Memory, it dont effect heaps directly.