Brackets are of the following three types :-
Parentheses '( )' Braces '{ }' Square Bracket '[ ]'

In a Mathematical problem, if there exists more than one type of Bracket, then the order in which we solve the problem is as :
First - Parentheses
Second - Braces
Third - Square Bracket

Working of all three types of brackets can be studied from the above respective links.