In 1835, John Willis, who operated a grist mill in Hempstead Harbor (now Roslyn) and (according to Francis Skillman) lived in the Caleb Valentine House, later owned by Augustus W. Leggett (T.G. 1977-1978), sold a number of house lots along the west side of the road which is today Main Street. According to the first published map of Roslyn, the Walling Map of 1859, the two houses identified today as #20 and #36 Main Street belonged to George Allen, a fish-merchant and horse-dealer. #36 Main Street, the George Allen Tenant House, has been much enlarged (T.G. 1978-1981). However, the earliest part of the house has many features in common with #20 Main Street, the George Allen Residence. The George Allen Residence is large by Village standards and has always been considered to be an important house.

In a letter to The Roslyn News (3/24/1883) describing life in Roslyn about 1850, M.A. Leggett, whose father, Augustus W. Leggett was copublisher of the Roslyn Plaindealer, at that time described it as "Allen's large white house", a statement which suggests that the house always was painted white. Its'
two-storey principal entry is architecturally the most important of its type in Roslyn.