Old restorative material should be removed if:

Old restorative material may remain on the pulpal or axial walls after initial tooth preparation. This material should be removed, if any of the following conditions are present: The old material may negatively affect the esthetic result of the new restoration (i.e., old amalgam material left under a new composite restoration). Radiographic evidence indicates caries lesion development under the old material and clinical evaluation confirms the radiographic interpretation. The tooth pulp was symptomatic preoperatively. The dentin along the periphery of the remaining old restorative material is soft. Retention of the existing material is compromised and the material is easily dislodged. If none of these conditions is present, it is acceptable to leave the remaining old restorative material to serve as a base, rather than risk unnecessary excavation in close proximity to the pulp, which may result in pulpal irritation or exposure.  Reference: Sturdevant operative dentistry. 7th EDITION, Page No:473