Imre was the son of Szemere Paksi de Paks but his mother is unknown. The date and place of his birth have not been found.
His wife was Ágota Héderváry de Hédervár. They were married, but the date and place have not been found. Their three known children were János (?-1526), Ferenc (?-?) and Balázs (?-1526).