This week kicks off one of the busiest travel periods of the year. Millions will set out for Thanksgiving, a holiday that celebrates how refugees from religious persecution found freedom in a new land. I’ll be one of those traveling. And while I’m looking forward to seeing family and friends, I’m not looking forward to the hassle: long flights, cramped seats, and the difficulty of finding a taxi at 3 a.m.

Of course, I’ll be travelling legally, with the right paperwork. Most of us can't imagine how much harder it is to cross international borders if you can’t get your papers in order. Yet millions of people fleeing war and instability, or fearing persecution from their own government, don’t have the luxury of applying for passports and waiting for visas. Without paperwork, they can’t move through border crossings legally. So they travel in other ways, often with no other choice but to resort to smugglers.

At Human Rights Watch, we’ve interviewed thousands of migrants and refugees over the past decades. People embark on complex, lengthy, and perilous journeys by foot, boat and plane – using false passports. They pay four to five times the price of the same trip by conventional means, selling a last piece of land or borrowing every last penny. For the refugees, “choosing” these journeys is a calculated risk weighed against even greater dangers at home.

The results are often deadly. Among the known fatalities in the last months: 12 Syrian refugees drowned off the coast of Egypt; 92 migrants, mostly women and children, died of thirst in the Sahara; and more than 300 people died when an overcrowded smugglers’ boat capsized off the coast of Italy. In 2012, 463 people died trying to enter the United States through harsh desert terrain.

Many of the children I have interviewed are unaccompanied – traveling without their parents or other guardians. At least 10,000 unaccompanied children claim asylum in the European Union each year, mostly from Afghanistan, while more than 4,000 unaccompanied children have fled Syria since the war began. The United States last year apprehended almost 25,000 unaccompanied children, most from Central America. Many report they are fleeing violence, poverty, and sometimes persecution and the threat of death. Unaccompanied children have no one to guide them through these calculated risks. They make decisions along their journey alone.

Arif fled Afghanistan alone when he was 15, hoping to seek refuge in Australia. When we met in Indonesia, he had twice paid smugglers to take him by boat to Australia – a crossing on which more than 1,000 migrants have died in the last 10 years. Both times he nearly drowned. On the second trip, he said he spent “three nights floating in the boat while it was sinking…We kept climbing higher and higher as the boat was sinking.”

Eventually rescued by the Indonesian navy, Arif was taken back to Java, where he was detained for months. He says he was beaten in detention. Still, Arif told me, he would try the boat trip again.

The smugglers’ “business” is unregulated, and there are no passenger manifests. So how many travel this way? Hundreds of thousands a year is a low estimate.

Many people in rich countries ask why some refugees travel so far. Why don't they stay closer to home? The truth is that many of the countries bordering conflict zones can’t or don’t protect refugees. Afghan asylum seekers entering Iran today are all but blocked from lodging refugee claims and are often deported back to Afghanistan. Between November 2012 and January 2013, Kenyan police detained, tortured, raped, and otherwise abused 1,000 Somali refugees. Although these and many other less-developed countries have generously hosted hundreds of thousands of refugees in the face of never-ending conflicts, their will and capacity is waning.

We don’t know how many of the people who travel this way have valid refugee claims. But the point is, we won’t know unless we let them ask. Desperate people need safe pathways to seek refugee status. Australia, the EU, and the United States can all do more to improve access to asylum. Countries as diverse as Indonesia, Pakistan, and Israel should be protecting refugees and should ensure that they are not abused during their journeys.

As our flights take us home, it’s worth thinking of the fragile boats that carried yesterday’s pilgrims and today’s refugees. We have a lot to be thankful for.