Member Avatar for Marymead-IT

Hi All,

I've been trying to set up a reliable system to send compressed videos of counselling sessions here (Australia) to a doctor in the U.S.

We had a VPN based system for a while, but for various reasons, this is no longer appropriate.

Transfer sizes are usually about 500MB - 2GB at a time.

I've set up an FTP server in the U.S. which uses a home-made SSL Certificate for data encryption. Security is reasonably important, but the likelihood of someone wanting to hack this is fairly low.


For some reason, the FTP sessions are dropping out whenever I try to send an individual file over about 300 MB.

If anyone has ideas for what could be causing these problems or suggestions for other solutions, that'd be great...

Regards,

Chris Fry

  • 6 Contributors
  • 9 Replies
  • 279 Views
  • 7 Months Discussion Span
  • Latest Post Latest Post by Marymead-IT

Dani AI

Generated

Files like counselling-session videos are likely to be regulated personal health information, so encryption and the legal rules for cross‑border disclosure need to be part of any design. U.S. HIPAA guidance treats encryption and cloud arrangements as required risk‑controls for ePHI and flags that cloud providers can be business associates; Australian APP 8 creates direct obligations when an Australian entity discloses personal information overseas. (hhs.gov)

The technical symptoms (FTPS with a home‑made cert, uploads consistently failing around ~300 MB) point to protocol and network issues rather than simple bandwidth. Classic FTP uses separate control and data channels, and FTPS/TLS can break firewall/NAT “FTP helpers” because the control channel is encrypted so middleboxes can’t rewrite ports — that often causes large transfers to stall. FTPS servers may also force TLS renegotiation thresholds that some clients don’t handle, dropping the data channel. (rfc-editor.org)

Practical next steps (in order of impact):

  • Move to SFTP/SSH or rsync-over-SSH where possible: SFTP runs over a single encrypted SSH channel and avoids the FTP control/data/NAT problems. Example resumable command: 
    rsync -av --partial --progress --append-verify -e ssh /path/largefile user@host:/incoming/

    Rsync supports robust resumption and verification. (manpages.opensuse.org)

  • If FTPS must be kept: configure a tight passive port range, set the server to advertise the external IP (masquerade), open those ports on any NAT/firewall, and disable/adjust forced TLS renegotiation per the server’s TLS settings. (sobek.hsdn.org)
  • Always encrypt at source (client‑side) before sending — for example GPG symmetric encrypt or use a client‑side crypt layer (rclone crypt) so only the recipient can decrypt. Example: 
    
gpg --symmetric --cipher-algo AES256 --output session.mp4.gpg session.mp4
``` ([manpages.ubuntu.com](https://manpages.ubuntu.com/manpages/xenial/man1/gpg.1.html?utm_source=openai))

Also check server disk quotas/filesystem limits, enable verbose server/client logs during a failing transfer, and test small staged transfers and resumption. Given the sensitivity of the material, document the chosen controls (risk analysis, encryption, access controls and any contractual safeguards such as BAAs) before regular operation.

Recommended Answers

All 9 Replies

Member Avatar for TheNNS

you have to adjust the settings on the ftp server so that it allows bigger files to be sent. i think you're also using way too much bandwidth, so this could be a problem also.

Member Avatar for Spriggan

First I would look into getting an SFTP server and client (secure file transfer protocal).

Then, I would use 7-zip, winRAR, or some other compression tool to compress these large files while simultaneously breaking them into smaller fragments that may have an easier time being uploaded/downloaded en mass.

Member Avatar for iam

First I would look into getting an SFTP server and client (secure file transfer protocal).

Then, I would use 7-zip, winRAR, or some other compression tool to compress these large files while simultaneously breaking them into smaller fragments that may have an easier time being uploaded/downloaded en mass.

What he said, 'cept for 2GB you shouldn't really need to break the archive up into segments, unless that is the doctor has a poor connection. But even then he just needs to use a download manager so downloads can be resumed.

Let us know how you get on :)

Member Avatar for Marymead-IT

Hi Guys, thanks for the replies.

TheNNS:
- The FTP Server has been configured to allow for unlimited file sizes
- Our internet connection is 2 Mbps, unlimited data transfers and the other end is faster, also with unlimited data transfers, so I don't think band width is the issue. However, it may be that one of our ISPs is timing out and dropping the connection.

Spriggan:
- I'll try archiving files, but they are already heavily compressed, so I won't get a smaller file out of it - i.e. original videos are in high definition, and roughly 25 GB/hour of footage. The other problem with compression/archiving is that it adds an extra, time consuming step to the process, but if this is the only solution, then we'll just have to do it.

I think "breaking them into smaller parts" could be a good solution, as long as its not too difficult to put them back together for our friend in the U.S. (his computer literacy is limited).

Iam: Do you know of a good, secure "download manager" solution that would be appropriate, or any open source projects I could modify to do this?

Thanks everyone for your help thus far.

Chris

Member Avatar for jbennet

um, why not just burn them to cd or dvd and mail them by recorded delivery ?

Member Avatar for TheNNS

> um, why not just burn them to cd or dvd and mail them by recorded delivery ?

it would take them quite a few dvd's, or a couple blu ray discs, (blu ray is way too expensive).

Member Avatar for lodenian

I use to send big files over. It's Free

Member Avatar for jbennet

They have no privacy policy - dont use them

Member Avatar for Marymead-IT

Thanks all,

jbennet - movies need to be sent weekly, and mailing from Australia to the U.S. is too time consuming/expensive.

We've got someone else working on this now, but I believe they're using an SFTP system.

Thanks for all your help,

Chris Fry

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.