Adding colliders changes the center of mass and rotation behaviour of a rigidbody. To set those manualy to the center of the object use:
void Start(){ rigidbody = GetComponent<Rigidbody>(); rigidbody.centerOfMass = Vector3.zero; rigidbody.inertiaTensorRotation = new Quaternion(0, 0, 0, 1);}