The problem here is that some versions of linux mint Cinnamon such as 17, 17.3 aren’t able to run without software rendering inside virtual box. In some cases even the live dvd won’t work without running in software rendering mode. Also for linux 17.3 with virtualbox 5.0.14 isn’t even able to run the OS GUI. It fails showing a message like, “x server failed” and switches to command line version.
Note that I can’t guarantee this solution will not work in your system but give it a try if there are no other options. The problem seems to be with linux mint 17 Cinnamon series and virtual box 5. Other free virtualization software such as VMWARE player works will work in case you need to use mint but, advanced feature such as screen recording may not be freely available.
1. Download latest linux mint version and setup normally in virtualbox. If it doesn’t work then proceed to next step.
2. Download VMWARE Player and install it.
3. Now setup mint iso with VMWARE player. Follow all necessary steps. Also no need to worry if it show that it couldn’t detect the OS.
4. Next start the virtual machine and from the machine install linux mint with default settings.
5. The vmware player should show notifications for installing vmware tools. First download the tools by clicking option in the notification. Then follow this tutorial to install vmware tools from terminal. Do this after installing mint on the virtual hard drive and not on the live dvd.
6. Now you should able to run linux mint fast and full screen in vmware player but if you still need virtual box then follow next steps.
7. Now inside virtual box click new and then click on expert mode to get the window shown below. From there follow all the necessary settings required for setting a virtual machine in virtual box. The only change is use an existing hard drive and select the vmdk machine created by vmware player. Typically located in document -> virtual machines folder.
8. If all the steps are followed correctly now you should have a fully working linux mint 17 virtual machine in virtual box. It should run the non software rendering version of cinnamon and support full screen workflow.
9. For the last step you may choose to install virtualbox guest additions. It will show a warning about replacing previously installed software without removing them properly might cause instability. I disregarded the message and installed anyway and It seems to be working for the moment.