#Gemma-4-31B-it

Gemma4ForConditionalGeneration is a dense multimodal model. This validation covers its text-generation serving path: the text backbone has 60 layers, hidden size 5,376, 32 attention heads / 16 KV heads, a 262,144-token vocabulary, and a native 262,144-token context window. The validated deployment is P4+D4: one 4-card Prefill engine and one 4-card Decode engine, coordinated by KServe LLMInferenceService and the InferNex Bridge. It is an 8 × Ascend 910B3 64 GB composite service, not a 4-card replica.

#Model identity

#Validated hardware × stack

#Model configuration

#Why P4+D4

The earlier aggregated TP4 deployment and P4+D4 are different resource envelopes. The aggregate service uses four NPUs and mixes long Prefill work with Decode; P4+D4 uses eight NPUs and removes Prefill dynamic steps from the Decode process. In the earlier capacity study, aggregate TP4 reached only 2.554 completed RPS on the 512/256 chatbot workload at target rate 3 over 900 seconds, with Running=104 and Waiting=294 at the end. P4+D4 is therefore the production candidate, but its throughput must never be presented as a four-card aggregate result.

#Deploy

The deployment spec is a self-contained KServe LLMInferenceService template with a TP4 Decode template, a TP4 Prefill template, InferNex router/scheduler, InferencePool, and the external route.

Before applying it, set the namespace, the two node names, the model PVCs, and registry references for your cluster. The manifest assumes the P/D pair has equal TP size and that the two engines can use Ascend Direct transport.

base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/plan/validated_models/assets/gemma4-31b-it
kubectl apply -f $base/gemma4-31b-it-pd-p4d4-llmisvc.yaml

# Internal KServe ingress (no MaaS API key):
curl -s http://<gateway>/<namespace>/gemma4-31b-it-pd/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"google/gemma-4-31B-it","messages":[{"role":"user","content":"hello"}]}'

#Benchmark results

Closed-loop aiperf 0.7.0, driven directly through the P/D proxy. Both scenarios run 480 requests per concurrency tier at concurrency 8 / 16 / 32, with warmup equal to the tier's concurrency. TTFT / E2E are seconds, ITL is ms, Decode is output-only token throughput, and TPS is input-plus-output token throughput.

Scenario ① — fixed-length system-prompt reuse (ISL 8,013 / OSL 128)

Scenario ② — multi-turn dialogue (average ISL 17,933 / OSL 128)

Every tier completed 480 requests with zero errors. Throughput and ITL are flat from concurrency 8 onward in both scenarios, while TTFT grows with concurrency. The service is Prefill-capacity-bound at roughly 8k input-dominated tokens/s: increasing average ISL from 8,013 to 17,933 lowers request throughput from 1.03 to 0.44 requests/s, while total token throughput changes by only about 5%. Decode ITL remains stable near 29 ms.