Jackson on Slavery
Andrew Jackson supported and participated in the institution of slavery. Not only did he own slaves himself, but he often stood against abolitionists, believing them to be a threat to national unity.
Jackson's support of slavery is a blemish on his record of standing up for the common man. Without even addressing the clear immorality of slavery, as an institution, it heavily benefited elite upper classes. It provided a cheap alternative to labor from free men (the people Jackson claimed to support), and its existence enabled plantation owners to continue building their wealth while spending next to nothing on labor. In dealing with slavery, Jackson still took numerous stands against the federal government, which was par for his course. However, instead of fighting on behalf of the common man, he fought for individual state rights. He saw state autonomy as the best way to protect the Union but did not properly recognize slavery as a tool of the elite.
While Jackson certainly did not seem to harbor any internal conflict on the morality of slavery, it did appear as if he believed he was acting with the Union's best interest in mind. He despised abolitionists and believed that they were in favor of dismantling the union.  When slavery did rear its head, Jackson sought to remove it from the public eye. For example, when some abolitionists began to spread anti-slavery tracts through the mail in 1835, Jackson supported a solution from Postmaster General Amos Kendall, who wished to grant southern state officials the power to ban the tracts. 
It cannot be determined whether or not Jackson's justifying slavery as necessary to the country's existence was reconciling some sort of internal conflict of his own. His participation in an institution that was built by and for the upper class is not becoming of a common man's hero.
Pro-slavery views were entrenched in the south, and it seems most likely that Jackson fell victim to this, rather than him having a secretly pro-elite mentality. However, he was an undeniable participant in an elite institution, and when slavery came to issue, his defense of the common man went by the wayside. Slavery is a mark against Jackson as a common man, though slavery's status as a divisive issue means that there are too many mitigating factors at play to reach a verdict on Jackson with this evidence alone.
 Brands, H. W. Andrew Jackson: His Life and times. New York: Anchor Books, 2006.
 Remini, Robert V. Andrew Jackson. Baltimore: Johns Hopkins University Press, 1998.