If you are on free plan, you might realize that you can’t upload any files larger than 100MB. This is the limit that cloudflare applies to Free plan. If you are on Enterprise , you might increase this limit to 500MB. Want over 500MB, you might need to ask cloudflare to make an exception.
Cloudflare is great, but 100MB is quite small for video upload. What solution could be?
– Upgrade to enterprise (oh no, i can’t afford it)
– Disable the proxy – you will not benefit from cloudflare any more
– Create a new direct url upload that will not go through cloudflare – it will needs some work and we will lose some protection.
What else you can do?
After Google, a solution just comes into the picture. YOu might notice that, if you upload a file less than 100MB, you can feel that it’s upload because it takes time to load. But if you upload any files larger than 100MB, you immediately got an exception. This tells me that Cloudflare blocks the upload before we actually sending the data.
How does Cloudflare knows that we are uploading more than 100MB ?
It’s actually simple, when we do a POST request, we send the content-length, cloudflare check this length and close the connection right away.
Now we know where Cloudflare do the check, so what if we remove the content length? Luckily, many webserver (apache, nginx) still handles the upload pretty well even if we don’t send Content-Length.
How we remove the content-length?
It depends on what language you are using (Javascript/Python/PHP..), check your upload library and see how we can tweak the header. I would like to share a Python script that i use to upload large files to my Immich sel-hosted photos/videos. If you don’t know Immich, google it.
from requests import Request, Session
import requests
import os
import time
from datetime import datetime
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
url = "https://photos.yourdomain.com/api/asset/upload"
file="My1GBVideo.mp4"
API_KEY="API KEY"
stats = os.stat(file)
headers = {
'Accept': 'application/json',
'x-api-key': API_KEY
}
data = {
'deviceAssetId': f'{file}-{stats.st_mtime}',
'deviceId': 'python',
'fileCreatedAt': datetime.fromtimestamp(stats.st_mtime),
'fileModifiedAt': datetime.fromtimestamp(stats.st_mtime),
'isFavorite': 'false',
}
files = {
'assetData': open(file, 'rb')
}
session = requests.Session()
request = requests.Request('POST', url, data=data,headers=headers,files=files)
prepped = request.prepare()
del prepped.headers['Content-Length']
r = session.send(prepped, verify=False)
print(r.text)