(?-imsx:\(([^\(]*?)\))

matches as follows:

NODE                     EXPLANATION
----------------------------------------------------------------------
(?-imsx:                 group, but do not capture (case-sensitive)
                         (with ^ and $ matching normally) (with . not
                         matching \n) (matching whitespace and #
                         normally):
----------------------------------------------------------------------
  \(                       '('
----------------------------------------------------------------------
  (                        group and capture to \1:
----------------------------------------------------------------------
    [^\(]*?                  any character except: '\(' (0 or more
                             times (matching the least amount
                             possible))
----------------------------------------------------------------------
  )                        end of \1
----------------------------------------------------------------------
  \)                       ')'
----------------------------------------------------------------------
)                        end of grouping
----------------------------------------------------------------------

C:\community>perl  calc.pl

(3 * 1) + 6 + 5+3
3 + 6 + 5+3
17.00