In the 1920s, Ernest James Brown (E.J. for short) settled in Spokane with his wife Myrtle (known as Theo). After opening a successful restaurant in 1927 called the Sawdust Trail on Sprague Avenue and Havana Street, E.J. and his wife embarked on a new business venture.
Beneath a marquee sign flashing “Dine-Dance, Dine-Dance” just off the old Sunset Highway (which later became Sprague Avenue), the Pirates Den became the hippest most hoppinest joint just outside the city limits of Spokane. Opened in 1929, it offered a dining room with enough seating for 350 people and a dance floor. Customers such as Sammy Davis Jr., Nat King Cole, and Louis Armstrong stopped by for food and a little jazz music when traveling through Spokane.
At some point in the 1930s, E.J. changed the name from the Pirates Den to the Harlem Club after it was said by some that it reminded them of the Cotton Club in Harlem. Like most such businesses during this period, the clientele of the Harlem Club was predominantly white. However, one night a week E.J. would open the club to the black community for special dance events.
In 1951, faulty electrical wiring caused a fire that destroyed the building entirely. It was a tremendous loss to both the white and black communities of Spokane. Brown and his family who had been living in apartments above the building moved just down the road, and many of his children were able to go to college because of the success of the Harlem Club.