"I knew that this film would challenge me in a way that scared me," the actress tells TheWrap