“We are talking about 60,000 people, but the number could climb to 80,000,” Interior Minister Anders Ygeman was quoted as saying by Swedish media.
He added on Wednesday the government has asked police and authorities in charge of refugees to organise their expulsion.
Sweden, which is home to 9.8 million people, is one of the European Union countries that has taken in the largest number of refugees in relation to its population. Sweden accepted more than 160,000 asylum seekers last year.
But the number of refugee arrivals has dropped dramatically since Sweden enacted systematic photo ID checks on travellers on January 4.
The mass expulsion was announced as Europe struggles to deal with a crisis that has seen tens-of-thousands of refugees arrive on Greek beaches with the passengers – mostly fleeing conflict in Syria, Iraq and Afghanistan – undeterred by cold, wintry conditions and deadly seas.
The UN says more than 46,000 people have arrived in Greece so far this year, with more than 170 people killed making the dangerous crossing.
Ygeman said the expulsions – normally carried out using commercial flights – would have to be done using specially chartered aircraft, given the large numbers, staggered over several years.
Swedish officials on Tuesday called for greater security at overcrowded asylum centres, a day after the fatal stabbing of an employee at a facility for unaccompanied youths.