This could be right person, age is correct:
1920 Census Baltimore City, Ward 15, #62
1532 Leslie St
both black, both MD,MD,MD
Mary Pindel, 35, Wd, laundress at home
Joseph Key, 52, Wd, plumber for a contractor

This is the right person:
1900 Census, Linganore, Fred Co, MD #19
all MD,MD,MD
Joseph Key 32, Sep 1867, 32, M 12 yr, farm laborer, owns home
Mary E Key 26, May 1874, 26, M 12 yr, total 6 children, 5 living
Roxie S Key 11, Oct 1888
Claonia A Key 9, Nov 1890
Chas C Key 5, Jun 1894
John W M Key 3, Nov 1896
Aubrey B Key 1, Sep 1898

1880 Census, Linganore, Fred. Co, MD, #169
Eliza Chambers, 58, cannot read or write
Joseph Key 21, laborer [born ca 1859 - 8 yrs off]
Mary C. Key 19, cannot write
E. Lincoln Key 10m

NOTE: #169 is
1880 Census, District 19, #173
173 Elias Chambers 57 laborer nn MD, MD, MD
Anna Chambers 55 nn MD, MD, MD
Adaline Chambers 27 dau nn MD, MD, MD
Julia Chambers 22 dau yn MD, MD, MD
Jennin Chambers 5 dau yn MD, MD, MD
JohnMaisy Chambers 7 gson MD, MD, MD
Malinda Chambers 3 gdau MD, MD, MD
Emma Chambers 9/12 aug gdau MD, MD, MD
Clifton Chambers 5/12 jan gdau MD, MD, MD
and #165 is Wm. H. Greenholtz (W)
1880 Census, District 19, #165
white family
all MD,MD,MD
WM. H. Greenholtz, 48, carpenter
Margaret L, 45
Harvey P, 12, palsied
Kenny E, 9
Gracie, 5
So, this is almost certainly the black community (Mt. Olive) just west of Sam’s Creek Road north of Rte. 31, just west of the Carroll County line. [RHS]

Thelma (Key) Lyles says her father (Charles C. Key) remembered attending Keys Chapel Church when he (Charles) was a little boy7

Only Joseph Key in area in 1870:
1870 Census, District 11, Carroll County, Md New Windsor PO, #301
all born MD, none can read or write
Dennis Key 52, farm laborer
Sarah Ann Key 35
John Key 14
Joseph Key 10 [born ca 1860]
Martha E Key 6
Sarah C Key 5/12 (Jany)
Perry Matthews 3

There is, however, Josephus Key, son of John Key Sr. Age fits better.
1870 Census Linganore, #448
John Key 68, works on farm
Ellen J Key 42
James W Key 18, works on farm
John H Key 14, works on farm
Mary L Key 10
Isaiah Key 7
Josephus Key 2 [born ca 1868]
BUT
In 1880 Josephus Key is listed as living with his parents, Rev John and Ellen Key in Old Fields
142 John Key 49 laborer MD, MD, MD
Ellen J Key 51 xx MD, MD, MD
Isiah Key 16 MD, MD, MD
Josephus Key ? MD, MD, MD
Wilson M Key 5 MD, MD, MD