Definitions of properly speaking:

• adverb:   in actual fact

  Example: "Properly speaking, they are not husband and wife"