malloc does not allocate beyond 65872

[AutoC]

Hi,
I need an array of approx. 2,00,000 elements.But malloc fails to allocate beyond 65872.
When I give
(short int*)malloc(sizeof(short int)*65000);
It works but seg faults when I give
(short int*)malloc(sizeof(short int)*66000);

whatsup with that?

[Tom Gunn]

What compiler do you use?

[AutoC]

gcc

[Salem]

Now why are you casting the result of malloc?

Could it be that you didn't include stdlib.h and thus the compiler considers malloc as returning an int rather than a pointer?
This can be a well hidden and very obscure bug.

> I need an array of approx. 2,00,000 elements.But malloc fails to allocate beyond 65872.
Is this the only malloc call in your program?
Is it really an array of short int's, as the casting suggests?

If any previously allocated memory was screwed up (say by a buffer overrun), then anything could be happening. Staring at one line of code which is perfect if run in isolation won't solve it.

[AutoC]

Now why are you casting the result of malloc?

Could it be that you didn't include stdlib.h and thus the compiler considers malloc as returning an int rather than a pointer?
This can be a well hidden and very obscure bug.

I need an array of approx. 2,00,000 elements.But malloc fails to allocate beyond 65872.

Is this the only malloc call in your program?
Is it really an array of short int's, as the casting suggests?

If any previously allocated memory was screwed up (say by a buffer overrun), then anything could be happening. Staring at one line of code which is perfect if run in isolation won't solve it.

I've included stdlib,no problem there.
Yes.It is an array of short ints
Here is the entire function

int WriteRIFF( char *waveFileName, short *waveform, long  numsamples, long samplingRate, int bytesPerSample)
{
        FILE *file;
        RIFFWaveFile *myRIFF = NULL;
        long i = 0;
        long size = 0;
        int result = SUCCESS;
        file = fopen(waveFileName, "w");
        myRIFF = (RIFFWaveFile *)malloc(sizeof(RIFFWaveFile));
        myRIFF->header = (RIFFHeader *)malloc(sizeof(RIFFHeader));

        strcpy(myRIFF->header->riffChunkID, "RIFF");
        myRIFF->header->riffChunkDataSize = numsamples*2 + 36;
        strcpy(myRIFF->header->riffType, "WAVE");


        strcpy(myRIFF->header->fmtChunkID,"fmt ");
        myRIFF->header->fmtChunkDataSize = 16;
        myRIFF->header->compressionCode = 1;
        myRIFF->header->numberOfChannel = 1;
        myRIFF->header->sampleRate = samplingRate;
        myRIFF->header->averageBytesPerSecond = bytesPerSample*samplingRate;
        myRIFF->header->blockAlign = 2;
        myRIFF->header->sigBitsPerSample = bytesPerSample*8;

        strcpy(myRIFF->header->dataChunkID,"data");
        myRIFF->header->dataChunkDataSize = numsamples*2;
        size = myRIFF->header->dataChunkDataSize/2;

/* This is where the problem is */

        myRIFF->data = (short int*)malloc(sizeof(short int)*size);
        for(i=0; i <size; i++)
        {
                myRIFF->data[i] = waveform[i];
        }

RiffWaveFile and RiffHeader are structures as follows.

typedef unsigned char UCHAR;
typedef short int INT;
typedef long DWORD;

typedef struct RIFFHeader
{
        UCHAR riffChunkID[4];           //0x00          
        DWORD riffChunkDataSize;        //0x04
        UCHAR riffType[4];              //0x08

        UCHAR fmtChunkID[4];            //0x0C
        DWORD fmtChunkDataSize;         //0x10
        INT compressionCode;            //0x14
        INT numberOfChannel;            //0x16
        DWORD sampleRate;               //0x18
        DWORD averageBytesPerSecond;    //0x1C
        INT blockAlign;                 //0x20
        INT sigBitsPerSample;           //0x22

        //@ This Element is not present for Uncompressed/PCM format
        //INT numberOfExtraBytes  __attribute__ ((packed));     //0x24

        UCHAR dataChunkID[4];           //0x24
        DWORD dataChunkDataSize;        //0x28

}RIFFHeader __attribute__ ((packed));

typedef struct RIFFWaveFile
{
        RIFFHeader *header;                             //0x00 to 0x2B
        short int *data __attribute__ ((packed));       //0x2C to dataSize

}RIFFWaveFile;

[aflneto]

well malloc is defined as

void *malloc(size_t size);

and size_t is probably 64kb is your computer.
use farmalloc.

http://vitaly_filatov.tripod.com/ng/tc/tc_000.78.html

this is a function that allows you to allocate more than 64kb
hope it helps

[AutoC]

that doesnt seem to be the problem. The input to the function short int *waveform was initialized in a similar manner.It didnt cause a problem.

[aflneto]

What I was trying to say malloc cant allocate that much memory. It can allocate a maximum of 64kb of memory in one go.
Hence why

(short int*)malloc(sizeof(short int)*65000);

works, but

(short int*)malloc(sizeof(short int)*66000);

fails
answering your original question.
farmalloc() function on the other hand allows you to allocate blocks larger than 64kb.

[Ancient Dragon]

As far as I can tell farmalloc() is only implemented on 16-bit compilers. There is no such thing as a "far heap" in 32 (or more)-bit programs because they use the flat memory model where all memory is considered near.

[aflneto]

I just realized how stupid I was being! also,farmalloc is a function implemented for windows as well. Sorry for not being helpful!

[Ancient Dragon]

int main()
{
short *ay = (short int*)malloc(sizeof(short int)*66000);
printf("%p\n", ay);
return 0;
}    //end of main function

The above worked without a problem using VC++ 2008 Express. Chalk another one up for Microsoft vc GNU :) (so far that makes two during the past seven days)

[aflneto]

int main()
{
short *ay = (short int*)malloc(sizeof(short int)*66000);
printf("%p\n", ay);
return 0;
}    //end of main function

The above worked without a problem using VC++ 2008 Express. Chalk another one up for Microsoft vc GNU :) (so far that makes two during the past seven days)

Actually the above also compiled on gcc without a problem. gnu won't go down without a fight. =P .

so there must be something else wrong with the code.

[Salem]

Regarding casting malloc.
Here's why it's a really BAD idea.
http://faq.cprogramming.com/cgi-bin/smartfaq.cgi?answer=1047673478&id=1043284351

Second, are these the ONLY malloc calls in your code, or are there others?
If you screw up with malloc even once, then any subsequent use of malloc/free can fail for any reason, at any time.
Staring at where it crashes will NOT tell you anything useful.

The only realistic chance that we (as remote viewers) have of telling you where such a problem is is by having the whole source code. If you've got masses of code, this is probably a non-starter. If you can trim it down to a couple of functions which still crash, then we could probably help.

Finally, let's talk about how you're corrupting memory (so my bet is you're doing this somewhere else where it would matter, only you don't realise it).

strcpy(myRIFF->header->riffChunkID, "RIFF");
myRIFF->header->riffChunkDataSize = numsamples*2 + 36;

The size immediately follows the ID.
Which is unfortunate, because the strcpy is going to copy FIVE bytes into space where there is only room for FOUR.
Fortunately, initialising the size takes place afterwards, so you get lucky.
But reverse the code, and your strcpy() would trash part of the size!

Maybe somewhere else, you're allocating a lot less that you thought (after all, \0 has overwritten some part of a size). Or maybe you're allocating a hell of a lot more than you bargained for, Either way, it isn't good.

BTW, which OS are you running gcc on?
If it's windows, then I would remark on your use of "w" when writing binary data.

[AutoC]

wow..ok..the malloc casting was out of habit.I'll stop doing that

Second, are these the ONLY malloc calls in your code, or are there others?
> The WriteRiff function is in a separate program that is included and its the only function called from that program. There are a lot of other mallocs but everything seems to work fine upto the WriteRiff call. I've verified it.
I'll fix the memory allocation :)

I'm running on fedora 10
I've attached the program that contains WriteRiff.Please take a look.

[Salem]

> I'm running on fedora 10
Excellent.

Then you can try either of these and see what's going on. valgrind ./myprog valgrind is a tool used for spotting all sorts of memory management problems.

gcc -o myprog prog.c -lefence
gdb ./myprog

electric fence (which you link with -lefence) is a malloc debugger. Should you overstep any malloc'ed memory, you will immediately get a segfault, and be dropped in the debugger at the exact line of code causing the problem.

[AutoC]

> I'm running on fedora 10
Excellent.

Then you can try either of these and see what's going on. valgrind ./myprog valgrind is a tool used for spotting all sorts of memory management problems.

gcc -o myprog prog.c -lefence
gdb ./myprog

electric fence (which you link with -lefence) is a malloc debugger. Should you overstep any malloc'ed memory, you will immediately get a segfault, and be dropped in the debugger at the exact line of code causing the problem.

Thanks.I'll try them both.

[AutoC]

I've somehow managed to fix that error through a sequence of attempts but cant figure out what fixed it. :| But the error was actually in another malloc. thanks to valgrind.

I'm stuck in another place and the command Im using is

fwrite( riff->header, 44, 1, file)

and valgrind says..
==32304== Syscall param write(buf) points to uninitialised byte(s)
==32304== at 0x55FAE3: __write_nocancel (in /lib/libc-2.9.so)
==32304== by 0x4F6925: _IO_file_xsputn@@GLIBC_2.1 (in /lib/libc-2.9.so)
==32304== by 0x4EC399: fwrite (in /lib/libc-2.9.so)
==32304== by 0x80546A1: WriteRIFFWaveFile (SphereInterface.c:375)
==32304== by 0x8054638: WriteRIFF (SphereInterface.c:359)
==32304== by 0x8049657: rmSilence (egySil.c:190)
==32304== by 0x804972D: main (egySil.c:213)

what does that mean?

[Salem]

I've no idea.
I stripped the code down to something simple, and it seems clean here.

/*-------------------------------------------------------------------------
 *  SphereInterface.c - Library of functions to read different wavefile formats
 *  Version:  $Name:  $
 *  Module:  
 *
 *  Purpose:  
 *  See:  
 *
 *  Author:  Hema Murthy ([e-mail])
 *
 *  Created:        Thu 01-Mar-2007 12:19:44
 *  Last modified:  Wed 13-Jun-2007 08:45:53 by hema
 *  $Id: SphereInterface.c,v 1.4 2007/06/14 08:25:49 hema Exp hema $
 *
 *  Bugs:  
 *
 *  Change Log:  <Date> <Author>
 *      <Changes>
 -------------------------------------------------------------------------*/

#include <stdio.h>
#include <string.h>
#include "stdlib.h"
#include "SphereInterface.h"

/*****************************************************************************
  function : WriteRIFF - write to a file in RIFF format assumes
             data is sampled at 16KHz and quantised to 16bits/sample
  Inputs   : wavefilename
  Outputs  : short waveform, numSamples
  **************************************************************************/

//=================== WriteWaveFile ===================//
int
WriteRIFF (char *waveFileName, short *waveform, long numsamples,
	   long samplingRate, int bytesPerSample)
{
  FILE *file;
  RIFFWaveFile *myRIFF = NULL;
  long i = 0;
  long size = 0;
  int result = SUCCESS;
  file = fopen (waveFileName, "w");
  myRIFF = (RIFFWaveFile *) malloc (sizeof (RIFFWaveFile));
  myRIFF->header = (RIFFHeader *) malloc (sizeof (RIFFHeader));

  strcpy (myRIFF->header->riffChunkID, "RIFF");
  myRIFF->header->riffChunkDataSize = numsamples * 2 + 36;
  strcpy (myRIFF->header->riffType, "WAVE");


  strcpy (myRIFF->header->fmtChunkID, "fmt ");
  myRIFF->header->fmtChunkDataSize = 16;
  myRIFF->header->compressionCode = 1;
  myRIFF->header->numberOfChannel = 1;
  myRIFF->header->sampleRate = samplingRate;
  myRIFF->header->averageBytesPerSecond = bytesPerSample * samplingRate;
  myRIFF->header->blockAlign = 2;
  myRIFF->header->sigBitsPerSample = bytesPerSample * 8;

  strcpy (myRIFF->header->dataChunkID, "data");
  myRIFF->header->dataChunkDataSize = numsamples * 2;
  size = myRIFF->header->dataChunkDataSize / 2;

  printf ("size:%d\n", size);
  myRIFF->data = (short int *) malloc (sizeof (short int) * size);
  for (i = 0; i < size; i++)
    {
      myRIFF->data[i] = waveform[i];
    }

  result = WriteRIFFWaveFile (myRIFF, size * 2, file);

  fclose (file);  /*!! was AFTER return */
  return result;
}

//=================== WriteRIFFWaveFile ===================//
int
WriteRIFFWaveFile (RIFFWaveFile * riff, int size, FILE * file)
{
  int result = SUCCESS;

  if (fwrite (riff->header, 44, 1, file) != 1)
    {
      result = FAILURE;
    }

  if (fwrite (riff->data, size, 1, file) != 1)
    {
      result = FAILURE;
    }

  return result;
}

int
main ()
{
  short data[] = { 1, 2, 3, 4, 5 };
  WriteRIFF ("wav.wav", data, 5, 123, 2 );
  return 0;
}

$ gcc SphereInterface.c
In file included from SphereInterface.c:24:
SphereInterface.h:31: warning: ‘packed’ attribute ignored
$ valgrind ./a.out 
==5800== Memcheck, a memory error detector.
==5800== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==5800== Using LibVEX rev 1804, a library for dynamic binary translation.
==5800== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==5800== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==5800== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==5800== For more details, rerun with: -v
==5800== 
size:5
==5800== 
==5800== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 11 from 1)
==5800== malloc/free: in use at exit: 62 bytes in 3 blocks.
==5800== malloc/free: 4 allocs, 1 frees, 414 bytes allocated.
==5800== For counts of detected errors, rerun with: -v
==5800== searching for pointers to 3 not-freed blocks.
==5800== checked 60,188 bytes.
==5800== 
==5800== LEAK SUMMARY:
==5800==    definitely lost: 62 bytes in 3 blocks.
==5800==      possibly lost: 0 bytes in 0 blocks.
==5800==    still reachable: 0 bytes in 0 blocks.
==5800==         suppressed: 0 bytes in 0 blocks.
==5800== Rerun with --leak-check=full to see details of leaked memory.
$ ls -l wav.wav 
-rw-r--r-- 1 user group 54 2009-08-26 19:42 wav.wav