President Trump announced on Twitter that his secretary of state was leaving his position and being replaced with CIA director Mike Pompeo.
The dramatic shake-up came during a delicate time for diplomatic relations, with direct talks agreed in principle with North Korea.
In a statement, Mr Trump thanked him for his service and wished his family well.
Why did he leave?
The news came just after Mr Tillerson cut short a trip through Africa, with a statement saying he returned a day early because of schedule demands in Washington.
Mr Tillerson reportedly disagreed with a number of the president's policies, including his recently announced proposal to impose tariffs on steel and aluminium imports.