I lurched into New York from Canada in the late 1970s, and provincial soot was still drifting off my old BMW when I realized that my knowledge of the city was deficient. I believed I had been the victim of a broadcast delay that left me with only the most out-of-date and watered-down information. In Canada, we understood New York as it appeared in The Apartment—a film that might as well have still been in theaters in Ottawa when I left. Apparently, the city had since slipped into its Taxi Driver phase without telling me.

Decades later, I know that even the timeliest of portraits wouldn’t have helped an outsider—especially a Canadian—understand New York. Like a sign on a shop door that reads, BACK IN FIVE MINUTES, everything written about this buzzing metropolis is bound to be at least partially untrue as soon as it’s committed to paper.

What makes living here so wonderful, and has enticed generation after generation of immigrants—ceaseless progress, paradox, romance, myth, and mystery—also makes secondhand appreciation nearly impossible. Really, there is no substitute for experiencing the city yourself, for putting down roots and watching things unfold in real time.

This is all a long-winded way of excusing myself for the countless misapprehensions of my early New York years.

Take Fifth Avenue, for example. I knew that the street’s midsection had been a seat of power for the robber barons of the Gilded Age. Not incidentally, Mark Twain, who gave these tycoons that label as an assessment of the era’s greed and excess, lived on the other end of the Avenue just north of Washington Square Park, very near where I live now. I was also aware that in more recent years, the upper reaches of Fifth Avenue served as a cradle for the rich.

But all of that was remote to an ordinary, fledgling New Yorker like me. To secure a foothold in the unfamiliar city, I chose to believe that the hallowed margins of Fifth Avenue mostly comprised what the street had become famous for—department stores, luxury shops, grand hotels, and the occasional office tower. You didn’t go there unless you needed to buy something or you worked there. Its residential character existed only for those few who held kingly fortunes. Ordinary, desperate, half-starved New Yorkers didn’t live on Fifth, and never would.

Writing this from my apartment in Greenwich Village, only slightly less desperate and starved than I was all those years ago, I find myself proved wrong by New York once again.

To appreciate Fifth Avenue is to first accept that it’s not a normal street. Conventional ideas of urban planning and zoning don’t apply to what is basically a six-mile-long asphalt contradiction. In its vastness, Fifth Avenue is an encapsulating metaphor for New York, stretching from Washington Square Park downtown to West 143rd Street in Harlem. It contains a little bit of everything—and, in some instances, a lot of everything—in continuously changing proportions: Retail and residential spaces mix along its shoulders; public and private spaces sit beside each other like cousins; shrines to art and woefully unartistic buildings scrape at each other’s sides; and, above all, it simultaneously functions as a dazzling center stage and a humble backdrop for New York life in all its forms.

Fifth Avenue’s primacy in Manhattan’s grid plan is as much a consequence of design as it is happenstance. Like much of the city, Fifth Avenue’s character was formed by deliberate northward expansion in the triumphant afterglow of the Revolutionary War.

Just before the turn of the 18th Century, New York’s Common Council was desperate to raise money and hired a Polish immigrant named Casimir Goerck to divvy up the city’s common lands north of 14th Street into equally-sized, sellable farm lots. In his survey, Goerck wisely planned for three north-south roads laid out in between the lots, allowing for uncomplicated travel and servicing. The road running down the center was called “Middle Road.”

A quarter century later, the Common Council realized that they were mucking the planning up and were ill-equipped to foster a burgeoning city on their own. The New York State Legislature—less hapless in those days—swooped in to help, appointing a commission who came up with a scheme that was as grand as its name: the Commissioner’s Plan of 1811.

The accompanying grid plan was largely based on Goerck’s surveys, and in fact the whole thing centered on Middle Road. The street’s name was changed to the less ominous-sounding Fifth Avenue, its purpose from a country road through farmland to a grand boulevard. It would cleave the island into two halves, and to this day, is the dividing line between east and west streets in Manhattan, with street numbers ascending in either direction.

By the middle of the 19th Century, Fifth Avenue’s promethean nature began to reveal itself. Titans of industry—people with names like Astor and Vanderbilt—realized that it wouldn’t be long before downtown Manhattan overflowed. In a seismic shift, they decided to stake a meaningful, central claim before the hordes caught on, moving north into farm country.

Their palatial homes would lend Fifth Avenue one of its enduring epithets: Millionaires’ Row. William Astor and John Jacob Astor III’s opposing townhouses on 34th Street were the first midtown mansions. There was Mary Mason Jones’s “Marble Row,” a series of extravagant chateau that dominated an entire city block and pushed the frontier for fashionable society up to 57th Street in the late 1860s. (She was the model for Mrs. Manson Mingott in her relative Edith Wharton’s novel The Age of Innocence.) The Vanderbilts, perhaps waiting to ascertain how much building would need to be done to upstage their neighbors, colonized the ten blocks south of Central Park in the late 1870s with three châteauesque mansions that constituted “Vanderbilt Row.”

But mansions alone don’t make a community, much less a city. The millionaires missed the infrastructure of wealth they’d left behind in the bustling center of commerce below 14th Street.

Strangely enough, the community building began with hotels. In 1859, the real estate investor Amos Richards Eno poured $2 million into building the 5th Avenue Hotel on 23rd Street. Though it was known as “Eno’s Folly,” being slightly disconnected from the city’s center, the hotel fast became a hub of polite society where everyone from Ulysses S. Grant to Boss Tweed conducted business. When the Prince of Wales stayed there on his American tour in 1860, the hotel’s reputation—and Fifth Avenue’s, as a metaphor for the well-heeled and posh—was cemented.

In the 1890s, embroiled in a furious family feud, the Astor offspring razed their neighboring townhouses and built opposing hotels: the Waldorf and the Astoria. The union of the grand renaissance structures could have been prefigured by the fact that they shared an architect, Henry Janeway Hardenbergh. When the Astors struck a truce in 1897, the hotels were connected by “peacock alley” and rechristened The Waldorf-Astoria. It was torn down and a new hotel was established on Park Avenue. In its place is the Empire State Building. The age of the Fifth Avenue hotel was officially born. The St. Regis, which still stands, followed in 1904; the Peninsula’s predecessor, the Gotham, in 1905; and The Plaza in 1907—also designed by Hardenbergh.

For the culturally inclined, few things are more intoxicating than a stroll down the stretch of Fifth Avenue known as “Museum Mile”. Beginning at 110th Street with The Africa Center and running south from there, the swath is dotted every few blocks or so with a different shrine to the arts. It could fairly be said that there isn’t a more diverse collection of museums on a single street anywhere else in the world: El Museo del Bario (105th), the Museum of the City of New York (103rd), the Jewish Museum (92nd), the Cooper Hewitt Museum (91st), the National Academy Museum (89th), the Guggenheim (88th), Neue Galerie (86th), and, at 82nd Street, the crown jewel, the Metropolitan Museum of Art.

Museum Mile is a limiting nickname, though, for it excludes one of the finer institutions of culture along Fifth Avenue: the Frick. After leasing the Vanderbilt house at 640 Fifth, Henry Clay Frick, an industrialist who found his vast fortunes in steel and railroads—the silicon chips and social networks of their time—decided it was high time to build a stately home of his own on the corner of 70th and Fifth. When it was finally completed in 1914, he lived among his collection of Old Master paintings—Vermeers, Rembrandts, Goyas—until dying five years later. In his will, he bequeathed the house and all its contents as a public museum. It is my belief that the collection is equally magnificent to, if less sprawling than, the Met’s.

Standing between 40th and 42nd Streets, it’s always difficult to think of a time when the New York Public Library hasn’t always been there. The Beaux-Arts building contains all the things you’d expect of a world-class library—a glorious reading room with 52-foot-high ceilings, archives of everyone from Virginia Woolf to Tom Wolfe, priceless medieval manuscripts, a maps division—and the twin lion sculptures who flank its entrance, Patience and Fortitude, lend the building a certain permanence and impenetrability. A little over a century ago, the Croton Distributing Reservoir stood in the very same spot. The structure’s 20 million gallons supplied the city with drinking water for 50 years, and Edgar Allen Poe, as well as many a Fifth Avenue chatelaine is said to have walked on its ramparts.

What’s gone from Fifth Avenue is almost as important as what remains. You’d be hard-pressed to find so much as a sign of B. Altman, the department store that opened at 34th Street in 1906, beside the old Tiffany & Co. flagship that moved there from downtown just a year earlier. Those two started a trend that would ensure Fifth’s reputation as the ne plus ultra of shopping streets: Lord & Taylor followed in 1914, Saks Fifth Avenue (whose very identity hinged on its location) in 1924, and Bergdorf Goodman in 1929. It wasn’t a coincidence that in later years, everyone from Armani to Zegna would open a flagship on Fifth.

One of the great paradoxes of Fifth Avenue is that its lack of a dedicated subway line makes it more popular, not less. There are stations along many of its cross streets, but none on Fifth itself. This characteristic is as much the will of Fifth Avenue’s prosperous early residents as the mansions that posed on its edge. Every attempt to construct a subway was met with vehement opposition because the people who lived there wouldn’t benefit from public transportation, and couldn’t stand the noise of trains rattling beneath their feet.

So, rather than descending into a subterranean corridor, far removed from identifying above ground features—architecture, parks, people—you’re obliged to confront and travel among them. In this way, Fifth Avenue makes itself unavoidable to residents and visitors alike.

And it sometimes seems as if the street is begging you to question its historicity. One can’t help but feel disbelief at the fact that Fifth Avenue itself predates everything else on it. The sprawling Central Park, dreamt up by Frederick Law Olmsted in 1857, was molded around the street rather than the other way around. The sensationally beautiful co-op buildings of architects like Rosario Candela, J. E. R. Carpenter, and McKim, Mead, & White defy time and rise out of the concrete like monuments to an ancient sophistication—most are less than 100 years old, though.

Perhaps the most telling detail about Fifth Avenue is that from 1908-1911, it was widened south of 60th Street. In the service of adding lanes to carry more traffic, the city demolished and truncated private mansions, gardens, stoops, and other “encroachments.” Even now, shops and buildings along the Avenue come and go with almost no ceremony. As much as Fifth has gradually come to reflect the people it exists to serve, it has taken on a life of its own. History, in a certain way, is irrelevant. It seems that no matter how you characterize Fifth Avenue, the only certainty is this: you can live in the city for years and never set foot on certain streets, but Fifth Avenue is not one of those. And if fortune is kind to you, you might even find yourself living on it.