Member Avatar for ankitdixit

Hello All, I am a software developer and got the opportunity to work on a large web development project. I want to know which source allow efficient data storage feature between file system and dbms in a large project. I have asked this query on Quora and according to this post, The use of a database management system is efficient because there are numerous techniques for storing and retrieving data but not an idea about file system. Can anyone use both of them? Any suggestions?

  • 4 Contributors
  • 3 Replies
  • 368 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by markanderson_1

Dani AI

Generated

To expand the thread for : the pragmatic, production-proven pattern today is hybrid — keep structured data and relationships in a DB, but push large unstructured payloads (images, video, archives) to an object store or filesystem and serve them through a CDN. Object stores are designed for cost‑effective, durable object storage at scale; CDNs give globally fast, cacheable delivery. Amazon S3 use cases. CloudFront for distribution and caching.

Databases can hold binaries, but be explicit about tradeoffs. PostgreSQL uses TOAST/large‑object mechanisms and will move big fields out‑of‑line; that helps but still makes backups, replication, and memory use heavier. Benchmarks and vendor guidance show DB reads/writes for large blobs are often slower and make operations (backups, replicas) bulkier. See PostgreSQL internals on TOAST and guidance on binary performance. PostgreSQL TOAST docs. Binary-data performance analysis (CYBERTEC). Frameworks and tooling also warn about larger backups and IO pressure from BLOBs. Prisma recommendations on storing large objects.

There are middle-ground and alternate choices. SQL Server offers FILESTREAM (BLOBs on NTFS with DB semantics) for large objects that need streaming access. FILESTREAM documentation. Document databases like MongoDB provide GridFS for >16MB files when keeping files and metadata together is important. GridFS spec. For scalable uploads, the common pattern is direct-to-object-store uploads using presigned URLs (upload to S3, then write metadata/URL to DB; reconcile on failure). Presigned POST/PUT docs (botocore/boto3).

Quick checklist to pick an approach:

  • If files are large or read‑heavy → object store + CDN.
  • If you need strict transactional atomicity for file + record → DB or a transactional hybrid (carefully weigh backup/replication cost).
  • If you already use a document DB and files are >16MB → GridFS.
  • If you need streaming APIs with DB control on Windows → FILESTREAM.

This expands on ’s pragmatic point, gives concrete platform options mentioned by , and clarifies the consistency/backups tradeoffs raised.

Recommended Answers

All 3 Replies

Member Avatar for Dani

You can definitely store data in the file system sometimes, and in the database other times. It really depends on the best tool for the job. Databases are great when you need to query big datasets. However, as an example of when the file system might be more appropriate, suppose you want to store data that doesn't change very frequently and is required on almost every page load. It would make sense to store that data in the file system on the web server, so it could easily be accessed on each page load, and eliminates the overhead of having to connect to, and then query, the database server (which is typically a different server than the web server on the network).

Member Avatar for pritaeas

Depending on what you need to achieve, you can also consider a document database or a graph database.

Member Avatar for markanderson_1

Hello,

Redundancy is the concept of repetition of data i.e. each data may have more than a single copy. The file system cannot control the redundancy of data as each user defines and maintains the needed files for a specific application to run. There may be a possibility that two users are maintaining the data of the same file for different applications. Hence changes made by one user do not reflect in files used by second users, which leads to inconsistency of data. Whereas DBMS controls redundancy by maintaining a single repository of data that is defined once and is accessed by many users. As there is no or less redundancy, data remains consistent. The file system does not allow sharing of data or sharing is too complex. Whereas in Database Management System (DBMS), data can be shared easily due to a centralized system.

Hope this helps :)
Mark Anderson

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.