In that case make sure you only remove the staged version, and add the file to your .gitignore to avoid making the same mistake a second time:
git reset filename # or git remove --cached filename
echo filename >> .gitingore # add it to .gitignore to avoid re-adding it