It’s hard to imagine Frank Sinatra’s career ever being in a slump, especially since today he is regarded as one of the greatest entertainers of all time. Nevertheless, Sinatra suffered through more than a few blows in the late ’40s and early ’50s that left his career in the dumps. However, a tumultuous marriage to Ava Gardner helped revive Sinatra’s career and provided him the inspiration needed to write one of the first concept albums ever created, In The Wee Small Hours, which he simply just called “the Ava Album”. 

Sinatra was king in the 1940s. The bobby soxers were obsessed with the Hoboken, New Jersey-born singer. They snuck into his dressing room, hotel rooms, and even into the trunk of his car. Sinatra-mania grew so strong that in October of 1944, his performance at New York’s Paramount Theatre was the cause of the Columbus Day riot. Sinatra was undoubtedly America’s first modern pop star. 

Despite his huge success, Sinatra’s career began to decline in the late 1940s into the 50s. He went from being polled as the #1 vocalist in 1941 to #4 in 1948 and was out of the top spots altogether in 1949. The bobby soxers’ interest in Sinatra waned as he entered his late 30s and the bad press surrounding his unabashed adultery and alleged mafia connections left many people with a bad taste in their mouths. To make matters worse, Sinatra’s publicist and close friend George Evans died of a heart attack in January 1950. Evans had been instrumental to Sinatra’s popularity and his death was devastating for the singer. 

It was during this career slump that Sinatra began to see actress Ava Gardner. Sinatra and Gardner first met years prior while she was married to her first husband Mickey Rooney and ran into each other a few times over the years, but it wasn’t until 1949 that Sinatra and Gardner began to see each other romantically.

Sinatra was still married to his first wife Nancy at the time, but he assured Gardner that he had left Nancy emotionally, physically, and geographically. Gardner fell madly in love with Sinatra, regardless of his marital status and the warnings her friends and his former lovers gave her. Despite their attempts to keep a low-profile, news of their affair soon broke and on Valentine’s Day 1950, Nancy officially announced that she was separating from Sinatra. After a long and expensive divorce preceding, Nancy and Frank Sinatra were officially divorced in October 1951, and on November 7, 1951, Sinatra and Gardner were married. 

“Everyone knew that Frank was at one of the lowest points in his career,” Gardner recalled in her book  Ava: My Story. “His golden voice was letting him down, hitting the more than occasional clinker; his marvelous phrasing was losing its smoothness. After years on top, he’d fallen to number five on the Downbeat vocalist poll, had gone quite a while without a bestselling record, and suffered the humiliation of having Metro switch his billing from No. 1 to No. 2 when On the Town with Gene Kelly was released.” 

Their love seemed doomed from the start. Both Sinatra and Gardner loved hard and were prone to fits of jealousy, especially as they were both magnets for attention from the opposite sex. They would have public fights before they were even wed and there were multiple occasions where Sinatra would threaten suicide if Gardner were to leave him. To top it off, Sinatra’s career was still at a low while Gardner’s was soaring. 

“I know now that Frank’s mock suicidal dramas – his desperate love singles to get me back to his side – were, at root, cries for help,” Ava explained. “He was down, way down. His contracts were being canceled. His wife’s lawyers were intent on screwing every possible dollar out of him, something that caused him to force a laugh and say, ‘Ava, I won’t have enough bucks to buy you a pair of nylons once they’re through with me.’ For Christ’s sake, he was a human being like the rest of us. He’d been the idol of millions and now he was being taunted as a washed-up has been. And Frank Sinatra was nothing if not a proud man.”

Gardner, one of the most popular actresses at that time, used her powers in Hollywood to get Sinatra an audition for the role of Maggio in the film From Here to Eternity. Not only did Sinatra land the role, but he did so well that he won an Oscar and Golden Globe in 1954 for “Best Supporting Actor”. This was a true turning point in his career. 

Slowly, Sinatra’s music career returned to success too. He had spent the last few years playing in empty venus and selling very few records – so few that Colombia A&R executive Mitch Miller said he couldn’t even give away a Sinatra album – but a meeting with Capitol Records vice president Alan Livingston in March of 1953 resulted in Sinatra signing a recording contract with the label. It was at Capitol that Sinatra began his work with conductor Nelson Riddle. Together they recorded “I’ve Got the World on a String” and Sinatra, who was typically not one to say much about his work, exclaimed “I’m back baby, I’m back!” 

Though Sinatra’s career was making a marvelous comeback, his marriage with Ava Gardner was crumbling. “Not even the great success Frank had in From Here to Eternity, the part that eventually won him an Oscar and totally revived his career, could help put us together again,” Gardner recalled. “Once things start to eat away at the facade of marriage – things like overhearing a hotel elevator boy tell your husband, ‘Oh, Mr. Sinatra, last time you were here it was with Miss X’ – once you lose your faith in what the man you love is telling you, there is nothing left to save.” 

After Gardner received a heartbreaking phone call from Sinatra announcing he was in bed with another woman, she decided it was all too much to bare and made the decision to leave him. On October 29, 1953, it was announced that the couple were separating and Gardner would be filing for divorce. It was a devastating moment for the both of them. But from this immense grief one of Sinatra’s most iconic albums, In The Wee Small Hours, was born. 

During his time at Capitol, Sinatra had the idea to record concept albums with Nelson Riddle. Sinatra and Riddle first recorded Songs For Young Lovers (1954) and Swing Easy! (1954) before Sinatra decided to make an album based on feelings of loneliness and lost love. By chance, Sinatra and Riddle were headed into Capitol Records’ office when they came across Riddle’s former roommate, Dave Mann. 

Mann told them he had just finished a song he was particularly proud of, so Sinatra and Riddle invited him into Capitol to play it for them. Upon hearing this song titled “In The Wee Small Hours of the Morning”, Sinatra knew that it was exactly what he was looking for. It was the perfect title track for his album and would serve as the anchor for all other songs selected for In The Wee Small Hours. 

The album’s 16 tracks were recorded in five sessions at KHJ Studios in Hollywood. Each session began at 8 pm and ended after midnight. Frank was adamant about making this album perfect. He would practice each song over and over again with pianist Bill Miller. “For Sinatra, a performance might be an evening, but a recording was forever,” author John Brady wrote in Frank & Ava: In Love and War. “He went over each song until every nuance, pause, and inflection had been totally mastered by him and the musicians as they laid down the tracks through February and early March of 1955.” 

Sinatra was so dedicated to this album that he left one studio session after only singing a few notes because he felt that his voice wasn’t quite right. “Now picture this: Sinatra comes in, and he’s got a felt hat on the back of his head, which was his trademark at the time,” musician George Van Eps recalls in Sessions with Sinatra: Frank Sinatra and the Art of Recording. “We were all set. Frank sang a few notes, stopped, then said, ‘Good night, fellas. I’ll see you tomorrow night.’ Walked out the door!”

“He came back the night next, and it was perfect,” Eps continued. “He was very picky about what he did.” 

Nelson Riddle even recalled in a 1982 radio interview that they went through “Last Night When We Were Young” thirty times before Sinatra was satisfied. 

Each song did exactly as Sinatra wanted them to do; listeners were transported into his world of loss and loneliness. It wasn’t just the songs themselves, but the way he sang them that allowed people to see themselves in his words. His voice had aged and deepened over the years, which allowed him to sound truly tired and sad. “It’s like a cello,” Nelson Riddle said of Sinatra’s voice at the time. “Ava taught him the hard way.” 

Sinatra’s commitment to these songs was plain to see. He was so emotionally connected to “When Your Lover Has Gone” that he openly wept when hearing the playback. This song in particular is noted as one of Sinatra’s most powerful performances on the album. 

Capitol Records was initially wary that the public wouldn’t love an entire album of such dark material, but they were proven wrong immediately upon the album’s release on April 25, 1955. In The Wee Small Hours received astounding critical appraise and landed at No. 2 on the Billboard 200 chart, remaining there for 18 weeks. 

“A public that had first been titilated, then offended, by the Gardner-Sinatra relationship, was now ready to recognize its validity once they heard it expressed as painfully and as poigiantly as this,” critic John Rockwell wrote for the New York Times. 

In The Wee Small Hours cemented Sinatra’s return to success. “This album served to separate Frank Sinatra from his contemporaries for all time,” Peter Levinson wrote in September in the Rain: The Life of Nelson Riddle. “He was at last accepted by the leading music critics as a truly formidable musical artist.” 

Today In The Wee Small Hours is regarded not only as one of Frank’s best albums, but as one of the greatest albums of all time. Rolling Stone gave the album a perfect 5 out of 5 score and ranked this album as #101 in their list of the “500 Greatest Albums of All Time”. AllMusic also gave In The Wee Small Hours a perfect score. 

This album also helped popularize the idea of concept albums. “Before this influential Sinatra record hit the shelves, songs were packaged and perceived, for the most part, as isolated poetic statements, or ‘singles’,” Andy Friedman wrote for the New Yorker. “By curating his albums with songs that told a story he wanted to tell, and by singing each word as if he wrote it, Sinatra introduced a level of personal expression to the recording process that reached literary heights.”

In The Wee Small Hours’ artwork itself was also very influential. Numerous album covers have been inspired by the iconic image of Sinatra standing in the street on a blue night, a lit cigarette in hand. 

Sinatra’s display of pain and loss in In The Wee Small Hours helped save his career, but it couldn’t help save his marriage with Ava Gardner. The two never lost love for one another and often tried to salvage their marriage, but it never worked.

“Every once in a while, Frank would call me in Madrid, London, Rome, New York, wherever I happened to be, and say, ‘Ava, let’s try again.’ And I’d say, ‘Okay!’ and drop everything, sometimes even a part in a picture,” Gardner recalled. “And it would be heaven. But it would never last more than twenty-four hours. And I’d go running off again, literally running. We could never quite understand why it hadn’t and couldn’t work out.” 

Though the two officially divorced in 1957, they always remained in touch and would, on occasion, still spend time together. In her book Ava: My Story, Gardner recalled how happy she was when Sinatra decided to visit her in Australia while she was filming On The Beach in 1959. 

“On a more positive note, my private life got a lift when the real Mr. Sinatra called and told me he was flying to Australia to see me,” she wrote. “What’s six thousand miles when you’re still in love?” 

The following year, Gardner gifted Sinatra a watch inscribed, “To Frank and desert nights, Ava” likely referring to their time together at Sinatra’s Palm Springs home. Sinatra would also send her a bouquet of flowers every year on her birthday. Gardner always kept these flowers after they died, lying them on a special place on her dresser until he sent a new bouquet the following year. (Ava Gardner Museum) 

Ava Gardner was the great love of Frank Sinatra’s life, and he was hers. Though their love story is one filled with hardship and tragedy, it is no doubt that the relationship was an influential piece of both of their lives. Without having loved Ava Gardner, Sinatra’s career may have never re-bloomed and he likely wouldn’t have felt the pain necessary to create what is regarded as one of the greatest albums of all time, In The Wee Small Hours.