Hi,
I am not sure if it is the right approach, but now it has worked. What i did is as below
On root folder for everyone group, i went to advanced add\remove rights and granted permission only on View Objects and not on sub objects.
Then I go to each folder and assign the user\group what ever access rights they need and when I see the user security for any folder, I see everyone group has no access on it