You would need to use validation toolkit to validate the file extension.
Use server side validation and chose file extesnion validation.
for the server variable, click the lightning bolt, in the form collection, chose the FileField Simple File Name.
For displaying errors, see the "Customizing Digital File Pro Error Messages" tutorial in the how to section of the DFP support page:
digital-file-pro/