Customising the Styles menu seems clear and easy to accomplish using the HTMLEditor interface. However as far as I and others can see it doesn't actually work, or at least is buggy (perhaps there are circumstances where it does work).
One solution is to edit this file:
directly (about line 205), but you won't be able to have different styles for different editors and your style could get overwritten if an update replaces the file.
As far as I can see it's best to avoid the STYLING tab in the Editor settings. It's puzzling to work out exactly what the problem is. In one example the styles did not appear because the XML produced from the CSS file had errors. It created <style name></style> instead of <Style name></Style> (XML is case-sensitive). Even when this doesn't happen, style attributes seem to be ignored. If this were fixed, parsing the CSS can only produce styles of limited use anyway. All styles will be applied with <span> tags so if you want to apply a style to float in image to the right for example, you won't be able to do this by clicking on an image and choosing a style.
Parsing the CSS won't be able to produce XML such as this, where the element is set to "img":
<Style name="Image on Left" element="img">
<Attribute name="style" value="float:left; margin-right:10px; border:solid 1px #000;" />
As I suggested I am not clear exactly what happens but I think the solution is to avoid this part of the interface, create the XML yourself and check the paths at the end of the /custom/yourpage.js file:
FCKConfig.EditorAreaCSS = ''; (make sure this is set to '' or your XML file will be overwritten)
FCKConfig.StylesXmlPath should be set to the XML file you've created.
Presumably this feature of the editor was tested so perhaps it does work in some cases, but not for me and a number of others. I think the feature would have to be re-thought if it were to be useful even if the bugs were fixed. Hopefully someone from WA support can clarify or correct some of these observations.