By 1915, she was so well-established that she became Alexander Stirling Calder's model of choice when he became Director of Sculpture for the Panama–Pacific International Exposition held in San Francisco that year. Her figure was "ninety times repeated against the sky" on one building alone, atop the colonnades of the Court of the Universe, roughly modeled on St. Peter's Square in the Vatican. In fact, Munson posed for three-fifths of the sculpture created for the event and earned fame as the "Panama–Pacific Girl".

In 1919 Audrey Munson was living with her mother in a boarding house at 164 West 65th Street, Manhattan, owned by Dr. Walter Wilkins. Wilkins fell in love with Munson, and on February 27 murdered his wife, Julia, so he could be available for marriage. Munson and her mother left New York, and the police sought them for questioning. After a nationwide hunt, they were located. They refused to return to New York, but were questioned by agents from the Burns Detective Agency in Toronto, Ontario, Canada. The contents of the affidavits they supplied have never been revealed, but Audrey Munson strongly denied that she had any romantic relationship with Dr. Wilkins. Wilkins was tried, found guilty, and sentenced to the electric chair. He hanged himself in his prison cell before the sentence could be carried out.

The Wilkins killing may even have marked the end of Munson's modeling career, although she continued to seek regular newspaper coverage. By 1920, Munson could not find work anywhere and was reported as living in Syracuse, New York, supported by her mother, who sold kitchen utensils door-to-door. In November 1920, she was said to be working as a ticket-taker in a dime museum.

In the summer of 1921 Munson conducted a nationwide search, carried by the United Press, for the perfect man to marry. She ended the search in August claiming she didn't want to get married anyway. On October 3, 1921 she was arrested at the Royal Theater (later the Towne Theater) in St. Louis on a morals charge related to her personal appearance with the film Innocence (the reissue title of Purity), in which she had a leading role. She and her manager, independent film producer Ben Judell, were both acquitted. Weeks later, she was still appearing in St. Louis, along with screenings of Innocence, enacting "a series of new poses from famous paintings".

On May 27, 1922, Munson attempted suicide by swallowing a solution of bichloride of mercury.

On June 8, 1931, Munson's mother petitioned a judge to commit her to a mental asylum. The Oswego County judge ordered Munson be admitted into a psychiatric facility for treatment on her 40th birthday. She remained in the St. Lawrence State Hospital for the Insane in Ogdensburg, New York, where she was treated for depression and schizophrenia for 65 years, until she died at the age of 104.