The original historical location of Brickfield Hill was the land south of Liverpool Street. As the clay was exhausted the brickfields expanded to the east, south and west.
The exploitation of clay resource to the east saw the expansion of the fields beyond Elizabeth Street by 1814 . Brickfield remains probably dating to the 1830s and 40s were found at Surry Hills
The brickfields also were to the west of George Street near Cockle Bay (Darling Harbour)
The move from Brickfield Hill
Unfortunately, the good fortune that accompanied the building boom and the growth in private brickmaking was dealt a serious blow with the closure of the government and private yards at Brickfield Hill.
Early locations of the Brickfield are seen as centering on the blocks between Campbell, Elizabeth, Liverpool and George Streets.
For more than 40 years they had provided most of Sydney's bricks, but by 1840 their presence was a hindrance to the rapidly growing metropolis. Industry and residential dwellings made uneasy neighbours, especially at a time when town planning was almost entirely lacking.
There was hardly a single tree in Hyde Park and the bare ground increased the intensity of violent windstorms – the 'brickfielders' – which swept up grit and dust in enormous volume. Quite apart from brickfielders and polluting kilns, Brickfield Hill and its hinterland were noted for crime, which was just as bad in 1840 as it had been earlier in the century.
The historian Geoffrey Scott refers to statements made by an elderly colonist at the turn of the twentieth century. The old man recalled that when he arrived in Sydney in 1838 barbaric sports still flourished around the pubs in Brickfield Hill.
'Horse racing, dog fighting, cock fighting, rat killing and prize fighting were all the rage', he related with a sort of nostalgic gusto. 'Scarcely anything else was thought of, and it was very hard for a young fellow to settle down and become steady'. In one Brickfield Hill tavern he saw two bulldogs fight, and when they had sunk their teeth into each other 'the owner of one dog chopped off two of his feet to show the breed of the dog that still kept on fighting!'  (www,dictionaryofsydney.com.au)
Such activities were considered perfectly normal ways for working men to indulge their passion for beer, a bet and some biff. Responding to complaints, the government stepped in, conscious of its responsibility to clean up areas that attracted such activity and the criminal behaviour that often came with it.
Towards the end of 1841 it ordered the cessation of brickmaking on Brickfield Hill, which had disastrous consequences for many people involved in the trade. One suspects the closure had less to do with cruelty to animals than it did with the merchants and well-to-do whose mansions and townhouses shared the same backyard as the industrial workings of Brickfield Hill, a 10-minute stroll from Hyde Park.
An Tale of an Incident at Brickfield Hill.
Speaking of convicts reminds me of the first time I came to Sydney when I was ten years old. At that time, there was a great trade done in cedar and my step-father (Mr Sykes) had arranged with Robert Dunn, who had a wheelwright's shop on Brickfield Hill, to build him a couple of carts, and to pay for them with cedar
We camped with a load ten miles out of Sydney, and for some unknown reason, my father was very anxious to get into Sydney early the next morning He woke up before daylight, and everything went well till we got as far as where the University now stands. There was a very bad road there, and we got stuck coming up the hill
However there was a chain gang working near at hand, and they gave us the necessary assistance, for which my step-father gave them a few figs of tobacco On top of the cedar and hiding it from view were several boxes of eggs and butter, and other farm produce, but on resuming our journey to Sydney, we found these had all mysteriously disappeared in the twinkling of an eye
My father sent to the overseer, and all the convicts were carefully searched, with no result except in one case One of the convicts had hidden a rooster underneath his jacket, and it was the noise of the fowl that proved his undoing.
The roads at the time were lined with a thick scrub, on both sides, and the convicts must have hidden the missing products there I now learnt the reason of my step-father being anxious to reach Sydney as early in the morning as possible
At that time it was necessary to have a licence to cut and remove cedar
Owing to the delay and there now being no covering for the load, Tom Dunn, the chief constable, was soon on our tracks, and coming up to my step-father asked him if he had a licence to sell cedar
The reply was 'No', whereupon Dunn took out a piece of chalk and drew a broad arrow on the cart, meaning by this that the contents were confiscated by the King
This was a pretty bad day's work, but luckily a man named Mooney, who had been formerly assigned to us, and was now in the employ of Dr Harris came up and asked my step-father what was the matter. Dr Harris lived in the locality which now bears the name of Homebush. Mooney went off post haste to Dr Harris (it is more likely that Mooney went to Dr Harris's Ultimo House, situated a short distance away in the area that has become in time the suburb of Ultimo.(To travel on foot to Homebush and back would have taken all day - or longer!), and returned in a short time with two letters: one to Dunn, and, in the event of that failing, another to be presented to Governor Macquarie.
Dr Harris in his letter to Dunn. indignantly demanded why he had interfered with his cedar. When Dunn opened it, he was in a terrible fright and immediately returned the cedar to my step-father, so there was no need to trouble the Governor in the matter." (Old Times , May, 1903, p.106). http://www.sykesfamily.com.au/william.htm