User-agent: *
Allow:/
Disallow: admin/ 
Disallow: download/ 
Disallow: includes/ 
Disallow: pub/ 
Disallow: media/