Member Avatar for Dani

What are some best practices for optimizing memory management when working with large datasets?

I am tagging this topic both with php (because that is my language of choice, and the one I work with big data with) as well as c++ (because I know DaniWeb has a large low level c++ community that is well suited to being able to delve into this topic into depth, and because years ago when I focused on c++ myself, I was very focused on efficiency).

  • 2 Contributors
  • 2 Replies
  • 101 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by Dani
seven7pillars commented: Great insights! Efficient memory management is crucial for handling large datasets. Techniques like data chunking, indexing, and using optimized data. +0
Member Avatar for Salem

My first thoughts would be

  1. What is large? Are we in the TB range or mere handfuls of GB?
  2. How often do you need to do this? Is it once a day, once a month, or just once.
Member Avatar for Dani

What is large? Are we in the TB range or mere handfuls of GB?

For the sake of argument, let's use my use case and say dozens of gigs and millions of rows.

How often do you need to do this? Is it once a day, once a month, or just once.

For me, the most important is real-time read and write performance to tables with millions of rows that have high concurrency.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.