Member Avatar for phorce

I'm working on a project that generates roughly between 13x512 values in which I compare at run-time. So the program outflow will be sort of as follows:

  • Program executes
  • Extracts (1) 12x512
  • Extracts (2) 12x512
  • Compares (1) to (2)
  • Extract (3)
  • Compares (3) to (2) (1)

The problem is, I don't know whether or not the quickest method would be to store the extracted values inside a text file and then read them in, or, just store inside memory and if I am storing inside memory, is there a quick way to access these? Such as memory mapping?

Thanks

  • 2 Contributors
  • 3 Replies
  • 222 Views
  • 53 Minutes Discussion Span
  • Latest Post Latest Post by Moschops
Member Avatar for Moschops

What's the size of each value? Are these simple int or double or the like, or some custom class?

If you're generating these values and then operating on them, writing them to a text file and then reading them back in again is massively expensive in time and pretty pointless. If you can hold them all in memory, do.

A memory map, as the term is commonly used in your scenario, is a way of quickly and efficiently working with a file of data, so if you're not using a file of data, it's not appropriate.

I'd suggest simply sticking the data in an array.

Edited by Moschops
Member Avatar for phorce

Hey, thank you for the reply. The values are just simple double values with nothing seriously data-intensive about them.

The program will be running for 20 minutes or so, depending on how long each recoding is for. So I can just dump the values that have not had any matches after a certain period. I just don't want to store massive amounts of data inside memory and for it to dramatically slow the program down?

Thanks

Member Avatar for Moschops

Things will only slow down if you've got so much data that you overwhelm your available RAM and the OS starts storing it on the hard drive in virtual memory.

Ignoring any optimisations like keeping data in a register or in the CPU cache, it takes the same amount of time to read data from anywhere in memory. Simply using a lot of RAM doesn't affect execution speed.

I'm sure there are various crazy edge cases and lots of riders on that like how much memory your OS likes to use before it starts using the HDD for extra storage and all that sort of thing, but 512 double values will take up on the order of 4 kilobytes. Given that a typical desktop PC has memory measured in the GB, 4 kilobytes is trivial. Using a millionth of your RAM for this data won't slow anything down.

Edited by Moschops
commented: Thank you for the help :) +7
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.