I set the output resolution too high and now the program crashes everytime I try to start it.

Download and install the latest release, and then reduce the output resolution. Newer versions allow you to change settings even if the renderer crashes, so you will be able to reduce the output resolution to a useable level.

Fractron 9000 crashes when I try to maximize it.

You might not have enough video memory to run fractron 9000 at your full desktop resolution. In the configuration screen, try using a custom output resolution that is smaller than your desktop resolution (such as 800x600).