Member Avatar for zyrus001

I'm reading in a file and sending the data (once encrypted) to a dictionary, with a hash of the data before and after encryption. I then pickle the dictionary but find the file size is massive compared to the source file size. If I write the encrypted data straight to a file the size is identical to the source. Any idea why my pickled file is so large?

#Encrypt data and get hashes        
    def encryptAndExportFile(self, key, inFile, outFile):
        
        openInFile = open(inFile,"rb")
        inFileSize = os.path.getsize(inFile)
        inFileData = openInFile.readlines()
        openInFile.close()
        
        """ initialise cipher """
        
        cipher = AES.new(key, AES.MODE_CFB)
        
        """ initialise MD5 """
        
        m = hashlib.md5() #hash
        h = hashlib.md5() #hash of encrypted dataq

        encryptedData = []
        
        for data in inFileData:
            
            m.update(data) 
            encData = cipher.encrypt(data)
            h.update(encData)
            encryptedData.append(encData)
            

        hashResult = m.digest()
        encHashResult = h.digest()
        
        return hashResult, encryptedData, encHashResult
def storeEncryptedObject(self, obj, path):
        
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile)
        outFile.close()
  • 2 Contributors
  • 2 Replies
  • 295 Views
  • 9 Hours Discussion Span
  • Latest Post Latest Post by vegaseat

Dani AI

Generated

Picking up from : the large pickle file is almost certainly caused by pickling a list of many small binary objects. Each list element becomes a separate pickled object with opcode overhead, so a list of N short encrypted chunks can balloon the file. Switching to a binary pickle protocol helps (as observed), but a better win is to avoid per-chunk object overhead entirely or to stream the data instead. is correct that requesting the interpreter's highest protocol is preferable when a pickle is required.

Practical options (prefer in this order):

  • Combine the encrypted chunks into a single bytes object before serializing; that collapses per-item overhead and keeps the saved size close to the raw encrypted size.
# store a single binary blob plus hashes
payload = {
    'md5': md5_digest,            # final digest bytes
    'enc_md5': enc_md5_digest,    # final digest bytes of encrypted data
    'data': b''.join(encrypted_chunks)
}
with open(path, 'wb') as f:
    pickle.dump(payload, f, protocol=pickle.HIGHEST_PROTOCOL)
  • Stream-encrypt and write directly to a file while updating hashes. This keeps memory use low and produces an output file that is essentially the same size as the encrypted file; store the two digests in a small header or a separate metadata file.
# streaming pattern (sketch)
h_plain = hashlib.md5()
h_enc = hashlib.md5()
cipher = AES.new(key, AES.MODE_CFB)
with open(infile, 'rb') as inf, open(outfile, 'wb') as outf:
    while True:
        chunk = inf.read(8192)
        if not chunk:
            break
        h_plain.update(chunk)
        enc = cipher.encrypt(chunk)
        h_enc.update(enc)
        outf.write(enc)
# record h_plain.digest() and h_enc.digest() separately

Notes and cautions: pickles are Python-specific and can be incompatible across Python versions or interpreters if using newer protocols; unpickling untrusted data is unsafe; encrypted data is not compressible (compression rarely helps and can even increase size); and MD5 is weak for cryptographic purposes — prefer SHA-256 for new integrity checks.

Recommended Answers

All 2 Replies

Member Avatar for zyrus001

Using protocol 2 greatly improved file size, which only increases by 5% in some cases.

def storeEncryptedObject(self, obj, path):
        
        outFile = open(path, 'wb')
        pickle.dump(obj, outFile, protocol = 2)
        outFile.close()
Member Avatar for vegaseat

Thanks for letting the rest of us know!
Most folks use protocol=-1 as the highest possible protocol.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.