This seems to be a fairly ambitious goal for an upload site. If your are able to handle all of the storage and bandwidth requirements though it seems like a cool idea.
I don't think that you would need to make a transaction to get it worked out. You would definitely want to log the uploads in your db though. You should have a users table and another table or more to handle the files. You should record the user that made the upload and the details about the uploaded file. You might want to have a joining table that can associate users with the file to keep the permissions straight. You can then have access rules to help you with the download server behaviors so that only the allowed users can get to the files.
You will also need a way for the uploader to specify who has permissions to the file. I think this would be workable by making use of DFP (or UE4 if you have it), Security Assist, and DataAssist.
