getgr*: validate group member lists from nscd backend

this corrects missing validation when using alternate group database
backends via nscd, as reported by 0rbitingZer0, which could result in
a heap-based buffer overflow.

while the source of truth for user (passwd) and group definitions is
generally an equal or higher-privilege domain than the application,
and compromise of nscd could inherently lead to bypass of some access
controls, it is still worthwhile to harden against direct attacks from
a compromised nscd.

this patch adds validation in the least invasive way possible,
erroring out at the point where a write past the end of the buffer
would previously have occurred.

a check is also added for member counts that would cause arithmetic
overflow in the existing buffer size computations, including negative
counts. this could be handled better by making adjustments where the
arithmetic is performed, but the way it's done here avoids making any
changes except for the actual bounds check.

src/passwd/getgr_a.c

@@ -71,6 +71,10 @@ int __getgr_a(const char *name, gid_t gid, struct group *gr, char **buf, size_t
 			goto cleanup_f;
 		}
 
+		if (groupbuf[GRMEMCNT] > (size_t)(INT32_MAX-1)) {
+			rv = ENOMEM;
+			goto cleanup_f;
+		}
 		if (groupbuf[GRNAMELEN] > SIZE_MAX - groupbuf[GRPASSWDLEN]) {
 			rv = ENOMEM;
 			goto cleanup_f;
@@ -127,7 +131,13 @@ int __getgr_a(const char *name, gid_t gid, struct group *gr, char **buf, size_t
 		if (groupbuf[GRMEMCNT]) {
 			mem[0][0] = *buf + groupbuf[GRNAMELEN] + groupbuf[GRPASSWDLEN];
 			for (ptr = mem[0][0], i = 0; ptr != mem[0][0]+grlist_len; ptr++)
-				if (!*ptr) mem[0][++i] = ptr+1;
+				if (!*ptr)
+					if (i<groupbuf[GRMEMCNT]) {
+						mem[0][++i] = ptr+1;
+					} else {
+						rv = EIO;
+						goto cleanup_f;
+					}
 			mem[0][i] = 0;
 
 			if (i != groupbuf[GRMEMCNT]) {