Classic Based Authentication :
Claim Based Authentication:
You cannot configure the Forms based authentication if your web application is using Classic Mode Authentication.
You can convert a web application from Classic Mode Authentication to Claims Based Authentication. However, that can only be done using PowerShell commands and its an irreversible process.
Classic authentication supports authentication types like Kerberos, NTLM, anonymous. Classic is more commonly seen in 2007 environments.
It enables authentication from windows as well as non-windows based systems. This also provides the capability to have multiple authentication in a single URL.
Claims based authentication uses claims identities against a against a trusted identity provider.
Claims are the recommended path for new deployments in SharePoint 2010