RANGER-5743: Docker setup for Apache Ranger ElasticSearch plugin and Integrate ES plugin with x-pack-security on ES 7.17

[pradeepagrawal8184]

What changes were proposed in this pull request?

This PR makes the Ranger Elasticsearch plugin work with Elasticsearch 7.17 + X-Pack Security instead of relying on client-supplied identity headers. It adds end-to-end Docker bring-up, fixes plugin packaging/classloading for ES 7.17, and wires audits through the Ranger audit ingestor path under Kerberos.

How was this patch tested?

The two commits were tested at three levels: automated unit tests (commit 1 only), manual Docker integration (commit 2, and validating commit 1 in a real ES stack), and problem reproduction before the fix. There is no new CI/integration test suite checked into the PR for the Docker or audit paths.

---

Commit 1 — RANGER-5740 (X-Pack caller identity)

Automated unit tests

Run with:

mvn -pl ranger-elasticsearch-plugin-shim verify

This was run repeatedly during development and passes today. It includes 5 JUnit 5 tests in two classes:

Test class	What it verifies
TestElasticsearchAuthenticatedUserResolver	Verified user comes from X-Pack SecurityContext; null when no user; system context skips auth requirement
TestRangerSecurityRestFilter	Rejects unverified Authorization: Basic (401, handler not called); accepts when X-Pack has set a verified user

These are mock-based unit tests — they do not start Elasticsearch or Ranger Admin.

Manual / planned testing

The commit message / PR test plan also listed manual checks that were not automated in the repo:

• Deploy on ES 7.17 with X-Pack Security enabled
• Authenticated REST requests authorized under the ES-authenticated user
• Wrong Basic password must not grant another user’s Ranger permissions

Pre-fix validation (motivation)

The original issue was reproduced separately on Ranger 2.9.0 in an isolated runtime replay: wrong Basic password still authorized as the named user. That confirmed the bug before this fix.

---

Commit 2 — RANGER-5743 (Docker + full ES 7.17 / X-Pack integration)

No new automated tests

This commit adds no new unit or integration tests. Validation was done by bringing up the Ranger Docker stack and exercising ES manually.

Documented in dev-support/ranger-docker/README.md:

mvn clean package -pl distro -am -DskipTests
cp target/ranger-* dev-support/ranger-docker/dist/
cd dev-support/ranger-docker
./download-archives.sh elasticsearch
docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-solr.yml \
  -f docker-compose.ranger-elasticsearch.yml up -d --build

Smoke tests from the README:

curl -u elastic:rangerR0cks! http://localhost:9201/test-index/_search   # expect 200 or policy 403, not 401
curl http://localhost:9201/test-index/_search                            # expect 401

Docker integration testing (manual, iterative)

The second commit was validated through many rebuild / recreate cycles of ranger-elasticsearch against a running stack (Ranger Admin, KDC, Solr, Kafka, audit ingestor, etc.). Issues found and fixed during testing included:

• Plugin tarball / .dockerignore / classpath / extended.plugins=x-pack-security
• REST wrapper conflict with X-Pack (removed Ranger REST wrapper)
• ES Security Manager / plugin policy / classloader init
• Policy cache path and seeding (ranger-elasticsearch-post-setup.sh)
• Internal ES users (_system, internal:*) bypass in RangerSecurityActionFilter
• Kerberos keytab + JAAS for audit posts to ingestor
• Ingestor AuditREST lazy allowed-users + dev_elasticsearch site XML

Authorization smoke tests (verified in Docker)

Against test-index on port 9201:

Request	Expected	Observed
Unauthenticated	401	401
elastic	200	200
testuser_2 (in Ranger policy)	200	200
testuser_denied (ES role OK, not in Ranger policy)	403	403

Test users are created offline via file realm (roles.yml + elasticsearch-users) because Ranger blocks ES security REST APIs.

Audit pipeline testing (partial)

Also tested manually in the same Docker stack:

• Sync audit client on ES request thread (ElasticsearchAuditIngestorClient)
• Kerberos SPNEGO to audit ingestor (not plain HTTP)
• ES logs showed successful posts, e.g.:
  • Audit event posted to ingestor for service=dev_elasticsearch user=testuser_2
  • Audit event posted to ingestor for service=dev_elasticsearch user=testuser_denied

Not fully verified end-to-end: audits appearing in Solr via Kafka/dispatcher (Solr Kerberos blocked direct query verification).

---