The man, thought to be 27, was detained as part of an intelligence-led operation on Parliament Street, according to the BBC.
A witness described seeing two knives on the ground, one of which he described as a large bread knife.
The Metropolitan Police said there were no injuries. The suspect is in custody in a south London police station.
A French tourist who saw the arrest said the suspect looked “very calm”. Another witness said the man was not shouting, or acting aggressively.
A police statement said: “The man was arrested on suspicion of possession of an offensive weapon and on suspicion of the commission, preparation and instigation of acts of terrorism. Knives have been recovered from him.
“Detectives from the counter-terrorism command are continuing their investigation, and as a result of this arrest there is no immediate known threat.”
The incident happened just yards from Downing Street, although Prime Minister Theresa May spent the day campaigning in Derbyshire and Leeds.
Mrs May said: “I think it shows that our police and our intelligence and security services are on the alert, as they always are, looking to keep us safe and secure.”
The BBC’s home affairs correspondent June Kelly said: “It’s understood that the man was detained as part of an ongoing operation by Scotland Yard’s counter-terrorism command.
“Officers working on intelligence moved in on him in Whitehall. He was then searched and arrested.”
It’s thought the man lives in London and holds a British passport, but was born outside the UK. Addresses linked to him are being searched.
Eyewitness Helen Chandler-Wilde described seeing “knives and a brown backpack lying on the floor” at the scene at the time of the arrest.
She told BBC Radio 5 Live that passers-by remained calm and were “remarkably not scared at all”.